https://prefect.io
Join Slack
Hi, Is there an easy way to move from a local agent with python scripts to an docker agent?
# ask-community
i
Hi, Is there an easy way to move from a local agent with python scripts to an docker agent?
k
Depends if you have custom dependencies. Do you have other modules you need to package?
i
what do you mean by modules?
k
Like files where you do
Copy code
from myfunctions import func
i
yes I have some
k
You need to package it in a Docker container like this
And then you can use Local storage with docker run like this
i
thank you, so first is to create the docker with all the requirements and my custom modules. and then I'll run the flow from my machine?
or I'll have to build a docker for each flow?
k
If you have many flows with all of the same requirements, you can just use the same image in the DockerRun. Just change the storage.
i
and running from my machine it will store the script in the dockker or it will launch the script with the modules in the docker
k
Docker agent will spin a new container for the Flow. If you use the image you built, it will have all the modules already. Does that make sense?
i
kind of... but maybe I'm not understanfing everything... you create the docker image with all the scripts? or how does it work. I'm not able to see from the local agent point of view
I mean whenever I want to add a new flow I just create a flow script and a tasks script and the register in prefect
but all of them run in the same machine with the local agent
so with docker if I create a new script will I need to create a new docker?
k
You have to rebuild the Docker image and put it in there, and then you Local Storage is a path inside the Docker container.
i
So I'l prepare a docker image with all the requirements and modules and add a workdir with all of my scripts cause they share the same modules
then How do I perform Flow.runs or Flow.register?
k
Same as before with 
flow.run()
. flow.run does not respect RunConfiguration and Storage. If you provide the image yourself, you can do:
Copy code
flow.register(build=False)
and it should be the same other than that
i
and I will execute it from my local machine?
k
If your docker agent is on the local machine then yes, it will spin a container on the local machine
i
no, I mean If I want to do a flow run will I do it from the docker or from my local machien?
k
If 
flow.run()
, it won’t use Docker. If registered flow, the agent picks it up and starts a new container to run the Flow in
I think you may as well just try it and it will make more sense
i
Ok I think now I get it more. Yes I think it will be easier just trying it out.
thank you very much Kevin!
👍 1
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by