https://prefect.io logo
Powered by
Title
a

Ankit

07/07/2020, 12:04 PM
Hey guys, I am new to the community and was trying to use prefect with docker. My use case being, I want to use the prefect server along with dask and run the same on my own ec2 instance using docker. I wanted to ask how to spin up the docker container for the same and when I have to register new flows to the server, I would want it to be as simple as adding the script to the src folder. I am a bit confused here and would appreciate the help. TIA
z

Zachary Hughes

07/07/2020, 12:58 PM
Hi @Ankit! There are a couple of items here, and I want to make sure we're unpacking/addressing them all. To start with, you want to run Prefect Server and a Dask cluster on the same EC2 instance, correct? If so, these two links should be helpful in getting started. https://docs.prefect.io/orchestration/tutorial/configure.html#spin-up-prefect-core-s-server https://docs.prefect.io/orchestration/execution/local_environment.html#using-a-daskexecutor-with-a-local-dask-cluster
Do you mind going into a bit more detail about your desired behavior regarding flow registration? In order for Prefect Server to have any notion of the flow, you will need to make an API call to server to register the flow.
a

Ankit

07/07/2020, 1:51 PM
Hey @Zachary Hughes, thanks for the quick response. I already have my docker-compose file for dask setup, was thinking of doing the same for prefect core(which would include prefect server and everything else). Any thing that could help me in that direction?
For the flow registration part, I think your suggestion to make an API call makes sense to me, will try that out once I reach that stage.
z

Zachary Hughes

07/07/2020, 1:59 PM
Are you trying to spin up a docker-compose file that has more than the one used by 
prefect server start
? I think I might be misunderstanding something about your use case.
a

Ankit

07/07/2020, 2:07 PM
Yes I have a separate docker-compose file, wherein all the dask specific services are defined. Was thinking of defining prefect as a service in that as well, by specifying the image and command 
prefect server start
inside it. So that, i'd only have to run 
docker-compose -f docker-compose.prefect.yml up
to run the entire thing. Sorry, if I am confusing you a bit here. 😅
A snippet of my compose file looks like this: 
version: "3"

services:
  scheduler:
    image: daskdev/dask:2.16.0
    command: dask-scheduler
    volumes:
      - scheduler:/tmp
    ports:
      - 8787:8787
      - 8786:8786
    environment:
      - DASK_DISTRIBUTED__SCHEDULER__WORK_STEALING=False
    networks:
      - dask
  prefect-service:
    image: "prefecthq/prefect:${PREFECT_SERVER_TAG:-latest}"
    command:
      - prefect server start
z

Zachary Hughes

07/07/2020, 2:27 PM
Okay, gotcha. I don't know that we have docs specifically for that, but what you're proposing sounds like it'll work with a bit of tweaking! The nice thing about docker-compose is that it's fairly modular, so I expect it'll work in a mostly plug-and-play fashion.
a

Ankit

07/07/2020, 3:18 PM
Yeah, you got that right. I am trying to tweak it already, just not sure if I am doing it right. Would be awaiting replies for your replies here as well. Thanks a lot @Zachary Hughes
I think I have hit a roadblock on the docker thing, since 
prefect server start
internally uses 
Dockerfile
and 
docker-compose
. So i guess i'll do with those commands for now. Another question @Zachary Hughes is, whenever my prefect container starts/restarts do I have to re-register all the flows that I had prior to the restart? Or is it saved somehow? If I have to then is there any option to register all the scripts present in a folder or something similar?
z

Zachary Hughes

07/07/2020, 3:56 PM
Since postgres is part of the docker-compose setup, restarting your prefect server setup will require re-registration by default. If you'd like to persist items on a separate volume, you should be able to use the 
--use-volume
flag when spinning up server for continued persistence. If you don't want to persist items between instances of Prefect Server, your best bet would be to write a script that registers each flow in a given directory with Prefect Server.
a

Ankit

07/07/2020, 3:58 PM
I was thinking about doing the custom script for it, not sure if it's an overkill though. Will explore the 
--use-volume
option you mentioned, see if it's helpful. Thanks a lot! 😃
z

Zachary Hughes

07/07/2020, 3:59 PM
Of course! Don't hesitate to reach out with any additional questions.
👍 1
#prefect-communityJoin Slack
Powered by