https://prefect.io logo
Join Slack
Powered by
I feel like this is a Noob question but I can't fi...
# ask-community
t
I feel like this is a Noob question but I can't find an answer. Is there a way to make prefect server restart automatically after reboots? The documentation mentions using supervisord for agents on this, but there don't seem to be any equivalent instructions for server. The closest I found was another thread on here that seems to use 
prefect server config
to generate a docker compose file, and I'm partway down the path of trying to set that up to auto-start (still working through https://github.com/flavienbwk/prefect-docker-compose, which was linked in it), but so far it's not working and it seems like there should be a simpler answer here. Docker has restart policies already (https://docs.docker.com/config/containers/start-containers-automatically/), is there not some flag I can pass to 
prefect server start
to use them?
a
Have you thought about deploying Server to Kubernetes using Helm chart? https://github.com/PrefectHQ/server/tree/master/helm/prefect-server this way you could offload this task of ensuring that all containers are running to Kubernetes
t
I haven't. I was originally going to try and set this up via the Azure Kubernetes Service, but it turns out I'd have to request enablement of that permission through IT... :-/ And the above looks cool but I'd also have to work through setting it, and Kubernetes up; is there no way I can just get the current, working, approach to auto-start? The above approach is likely better but at this point I just want to get to a working version faster, and it feels like I'm just one step away.
k
I think the approach is right in using flavienbwk’s repo and doing 
docker-compose up
upon reboot. What happens when you do 
prefect server start
was a startup script? Although of course this won’t revive containers that die while the machine is alive (maybe you can combine it with a restart policy?)
t
Hmm, okay, sounds like I was going down the right path at least. It seems like there are differences though, because if I do 
prefect server config > docker-compose.yml
and then do 
docker-compose docker-compose.yml up
, the UI doesn't load at the end. Are there other steps or differences there which I'm missing?
@Anna Geller, haha, you're paid to give that answer so I can't begrudge you that. 🙂 And honestly if I was doing this for myself I'd have made the switch already (and might soon for my personal projects at home). My point though is that I've basically worked through everything except this step already to get to functional, at which point I could start transitioning all my other ETL processes over to this instance and let it run for a few weeks to test it. In your above list, the only concern I have at this point is keeping the server up. Everything else I've addressed by having it inside a VNET or using supervisord, so I actually think I can use this in production. Once I do, I can set up flows to check for failures and rerun (dealing with network failures is one of my main drivers here), and if that works I could then use that track record, and things like RBAC, to justify starting a Prefect Cloud process.
k
No no we’re not paid to give that answer 😅. We’re paid to help the community including server users. Just that it’s a non-trivial problem as you’ve noticed. Most production server users are indeed on k8s like Anna suggested and heavily maintaining it (clearing out some database tables regularly at the very least).
upvote 1
Have you seen this ? I think this might cover the container crashing too
t
Hmm, but in that case, why doesn't my compose up command load the UI? What's the difference between the start command and that? Think I'll need to fix that before the docker restart stuff helps me.
k
The start command generates the config and then executes it. As to why the UI is not loading is hard to guess. Is there any clue when you look at the container logs?
a
hahaha 😄 Gotcha. Yeah, I understand the restrictions. So you have your own VNet and inside of it you have installed prefect and started Server + an agent with supervisor, and now the problem is to ensure that the processes that were started with “prefect server start” keep running, correct? I think the UI problem may be due to missing expose flag?
Copy code
prefect server start --expose
t
Also, I didn't mean the paid comment as a critique, hope it didn't come across that way @Anna Geller. Pointing out where your product is better than what I'm trying seems like a perfectly reasonable thing to do.
@Anna Geller, actually, I have the expose flag in there. I was being imprecise with my comments earlier, what I actually ran was 
prefect server config --expose > docker-compose.yml
, and I checked, doing so set the IP in several places to 0.0.0.0 as it should. When I run the 
docker up
command on the result though the UI doesn't load.
a
I see. The entire “prefect server config” afaik was meant as a replacement for “docker-compose up” so that when you use it, you no longer need to use docker-compose and set everything via prefect CLI: https://github.com/PrefectHQ/prefect/blob/57dbb113114603a743bec2fc407307336a9f2385/src/prefect/cli/server.py#L174 But there is no option in the CLI to run it as a deamon process
@Tom Shaffner I actually didn’t know about this approach with storing the config to a YAML file - this should actually work just fine. At least it worked locally for me:
Copy code
prefect server config --expose > docker-compose.yml
docker-compose up -d
You can later track the health (manually though):
but in theory, failed components should restart because of:
Copy code
restart: always
t
Oh!! So always IS on! That would indicate the failures I'm getting are likely issues with docker restarting, not the server.
I'm still not sure the docker file I'm getting is the same otherwise though. If I run the above vs. start a python environment and run: 
from prefect.cli import server
server.setup_compose_file()
I get pretty different outputs, and the only difference should be the IP I think. I haven't gone through it all yet, but is there nothing in the stripped-down config version that's missing? In the meantime I'll see about docker restarting after a reboot; maybe that's my problem if always is defaulted to on.
a
is there nothing in the stripped-down config version that’s missing?
based on my local test, all components were spun up correctly. So it looks like this is what you need. Perhaps you can compare your docker-compose with the one from here to see if there is something else you want to include, e.g. an agent https://github.com/flavienbwk/prefect-docker-compose
t
I think that did it! 
sudo systemctl start docker
sudo systemctl enable docker.service
sudo systemctl enable containerd.service
Then 
prefect server start --expose -d --use-volume
, and I now seem to be persisting beyond a reboot without restarting the DB each time. Combine that with the supervisord instructions you have up and I think I'm up and running!
a
Nice work! Btw, you could also attach a volume within a docker-compose file, so both should work.
👍 1
t
Agreed; for simplicity I'm just doing local runs for now, but keeping my reqs in a DOCKERfile in preparation for eventual upgrade/migration.
👍 1
201 Views
Open in Slack
PreviousNext
Powered by