Hey guys, I am new to the community and was trying...
# prefect-community
Hey guys, I am new to the community and was trying to use prefect with docker. My use case being, I want to use the prefect server along with dask and run the same on my own ec2 instance using docker. I wanted to ask how to spin up the docker container for the same and when I have to register new flows to the server, I would want it to be as simple as adding the script to the src folder. I am a bit confused here and would appreciate the help. TIA
Hi @Ankit! There are a couple of items here, and I want to make sure we're unpacking/addressing them all. To start with, you want to run Prefect Server and a Dask cluster on the same EC2 instance, correct? If so, these two links should be helpful in getting started. https://docs.prefect.io/orchestration/tutorial/configure.html#spin-up-prefect-core-s-server https://docs.prefect.io/orchestration/execution/local_environment.html#using-a-daskexecutor-with-a-local-dask-cluster
Do you mind going into a bit more detail about your desired behavior regarding flow registration? In order for Prefect Server to have any notion of the flow, you will need to make an API call to server to register the flow.
Hey @Zachary Hughes, thanks for the quick response. I already have my docker-compose file for dask setup, was thinking of doing the same for prefect core(which would include prefect server and everything else). Any thing that could help me in that direction?
For the flow registration part, I think your suggestion to make an API call makes sense to me, will try that out once I reach that stage.
Are you trying to spin up a docker-compose file that has more than the one used by
prefect server start
? I think I might be misunderstanding something about your use case.
Yes I have a separate docker-compose file, wherein all the dask specific services are defined. Was thinking of defining prefect as a service in that as well, by specifying the image and command
prefect server start
inside it. So that, i'd only have to run
docker-compose -f docker-compose.prefect.yml up
to run the entire thing. Sorry, if I am confusing you a bit here. 😅
A snippet of my compose file looks like this:
Copy code
version: "3"

    image: daskdev/dask:2.16.0
    command: dask-scheduler
      - scheduler:/tmp
      - 8787:8787
      - 8786:8786
      - dask
    image: "prefecthq/prefect:${PREFECT_SERVER_TAG:-latest}"
      - prefect server start
Okay, gotcha. I don't know that we have docs specifically for that, but what you're proposing sounds like it'll work with a bit of tweaking! The nice thing about docker-compose is that it's fairly modular, so I expect it'll work in a mostly plug-and-play fashion.
Yeah, you got that right. I am trying to tweak it already, just not sure if I am doing it right. Would be awaiting replies for your replies here as well. Thanks a lot @Zachary Hughes
I think I have hit a roadblock on the docker thing, since
prefect server start
internally uses
. So i guess i'll do with those commands for now. Another question @Zachary Hughes is, whenever my prefect container starts/restarts do I have to re-register all the flows that I had prior to the restart? Or is it saved somehow? If I have to then is there any option to register all the scripts present in a folder or something similar?
Since postgres is part of the docker-compose setup, restarting your prefect server setup will require re-registration by default. If you'd like to persist items on a separate volume, you should be able to use the
flag when spinning up server for continued persistence. If you don't want to persist items between instances of Prefect Server, your best bet would be to write a script that registers each flow in a given directory with Prefect Server.
I was thinking about doing the custom script for it, not sure if it's an overkill though. Will explore the
option you mentioned, see if it's helpful. Thanks a lot! 😃
Of course! Don't hesitate to reach out with any additional questions.
👍 1