https://prefect.io logo
Title
i

iñigo

05/18/2022, 2:42 PM
Hi, Is there an easy way to move from a local agent with python scripts to an docker agent?
k

Kevin Kho

05/18/2022, 2:44 PM
Depends if you have custom dependencies. Do you have other modules you need to package?
i

iñigo

05/18/2022, 2:45 PM
what do you mean by modules?
k

Kevin Kho

05/18/2022, 2:45 PM
Like files where you do
from myfunctions import func
i

iñigo

05/18/2022, 2:46 PM
yes I have some
k

Kevin Kho

05/18/2022, 2:49 PM
You need to package it in a Docker container like this
And then you can use Local storage with docker run like this
i

iñigo

05/18/2022, 2:54 PM
thank you, so first is to create the docker with all the requirements and my custom modules. and then I'll run the flow from my machine?
or I'll have to build a docker for each flow?
k

Kevin Kho

05/18/2022, 2:55 PM
If you have many flows with all of the same requirements, you can just use the same image in the DockerRun. Just change the storage.
i

iñigo

05/18/2022, 2:57 PM
and running from my machine it will store the script in the dockker or it will launch the script with the modules in the docker
k

Kevin Kho

05/18/2022, 2:58 PM
Docker agent will spin a new container for the Flow. If you use the image you built, it will have all the modules already. Does that make sense?
i

iñigo

05/18/2022, 3:00 PM
kind of... but maybe I'm not understanfing everything... you create the docker image with all the scripts? or how does it work. I'm not able to see from the local agent point of view
I mean whenever I want to add a new flow I just create a flow script and a tasks script and the register in prefect
but all of them run in the same machine with the local agent
so with docker if I create a new script will I need to create a new docker?
k

Kevin Kho

05/18/2022, 3:05 PM
You have to rebuild the Docker image and put it in there, and then you Local Storage is a path inside the Docker container.
i

iñigo

05/18/2022, 3:08 PM
So I'l prepare a docker image with all the requirements and modules and add a workdir with all of my scripts cause they share the same modules
then How do I perform Flow.runs or Flow.register?
k

Kevin Kho

05/18/2022, 3:10 PM
Same as before with
flow.run()
. flow.run does not respect RunConfiguration and Storage. If you provide the image yourself, you can do:
flow.register(build=False)
and it should be the same other than that
i

iñigo

05/18/2022, 3:11 PM
and I will execute it from my local machine?
k

Kevin Kho

05/18/2022, 3:11 PM
If your docker agent is on the local machine then yes, it will spin a container on the local machine
i

iñigo

05/18/2022, 3:40 PM
no, I mean If I want to do a flow run will I do it from the docker or from my local machien?
k

Kevin Kho

05/18/2022, 3:42 PM
If
flow.run()
, it won’t use Docker. If registered flow, the agent picks it up and starts a new container to run the Flow in
I think you may as well just try it and it will make more sense
i

iñigo

05/18/2022, 4:23 PM
Ok I think now I get it more. Yes I think it will be easier just trying it out.
thank you very much Kevin!
👍 1