https://prefect.io logo
Title
a

Anish Chhaparwal

10/05/2020, 5:50 PM
hey, I want to deploy my flow using docker. I'm using the code from prefect docs : Flow with docker (first image attached). I do not push it to any registry as i want to run the docker image on my local machine. After starting the docker agent using
prefect agent start docker
i am able to register the flow to the UI but it always remains in pending state and never gets executed(refer images attached). Can someone explain where i am going wrong?
@Kyle Moon-Wright hey i was reading your reply and somehow i cant see it anymore.
k

Kyle Moon-Wright

10/05/2020, 6:45 PM
Whoops, @Anish Chhaparwal looks like I was wrong about my initial understanding. It looks like your container is starting up but can't communicate back to your server instance because it communicates with cloud.prefect.io by default, so your flow will continue to be rescheduled via the Lazarus process. In this case, you should pass an
--api
flag to your agent so it looks to your server instance i.e.
prefect agent start docker --api <your_endpoint>
Give the api flag on your agent a shot. you should be able to use local images w/o registry no problem.
a

Anish Chhaparwal

10/05/2020, 6:51 PM
when you say --api <your_endpoint> are you referring to graphql?
k

Kyle Moon-Wright

10/05/2020, 6:56 PM
Yes, your Prefect Server Graphql Endpoint which defaults to http://localhost:4200/graphql once you have your instance of server running.
a

Anish Chhaparwal

10/05/2020, 7:13 PM
@Kyle Moon-Wright Yes it did work. thank you for your quick response!! I have a couple of basic questions... 1. Our team is developing a couple of flows on a server. Every time we run a flow does flow.storage = Docker() use the existing docker image and just create a new container every time we run it? which would mean different users are developing in the exact same environment? is my understanding correct? 2. how do we tell the flow which docker image to use if multiple prefect docker images are present on local machine? 3. when using prefect do we ever need to run commands from within the container? 4. if multiple agents are run on the same server we deploy them with different labels. Right before registering a flow, is using the statement : flow.environment = LocalEnvironment(labels=["your_label_name"]) correct way to tag a flow to a particular label?
k

Kyle Moon-Wright

10/05/2020, 7:57 PM
First off, it's worth noting that Prefect Server was not intended for production nor team collaboration, but moreso for single developers and hobbyists - anything beyond and our recommendation is Prefect Cloud. That being said: 1. Using this storage method without any configuration, every time you run the script you will create a new local image and create a container to execute. To collaborate, I'd recommend putting that shared image in a registry others can access to ensure it's the same image. 2. You can specify which image you'd like to use by using the Docker storage kwargs, I believe you'll need the
local_image
and
path
kwargs for you local configuration (I'm not deeply familiar with using local images). 3. You can if you need to! The command kwarg would be useful in this case, but is not necessary. 4. Yep, I find that's the easiest way to attach labels - though you can also attach labels through the UI or the API.
a

Anish Chhaparwal

10/07/2020, 5:27 AM
I have a lot more clarity now. thanks for that. let's say the shared registry url is:
registry_url="<http://gcr.io/dev/|gcr.io/dev/>"
how do i use the same image on a new flow? If i import docker and create a docker instance wont it a new image (based on shared image) for each flow?
k

Kyle Moon-Wright

10/07/2020, 6:25 PM
You'll need to configure your flows to use the custom base image that you built and configure a Dockerfile like in the example here.
a

Anish Chhaparwal

10/08/2020, 2:04 PM
seen the documentation before but was a little confused. will ping if i get stuck. thank for being patient! :)