Frank Hereford
05/17/2022, 2:58 PMemre
05/17/2022, 3:16 PMFrank Hereford
05/17/2022, 3:29 PMflow.register()
method is called, the individual tasks are compiled into python bytecode and pickled using cloudpickle
. This frozen representation of the compiled tasks is sent over the network up to Prefect’s cloud service. (We’re a subscriber, if that is helpful information.)
At the same time, I have an agent running. This agent runs from the same docker image, and in fact is the entry point for the container. In this way, the docker image serves both as a development environment and the environment for the agent to run. When the flow is to be kicked off as per the schedule or a one-off execution, the compiled bytecode is sent from the prefect cloud service down to the agent, where it is executed. This bytecode expects any library or package dependencies as defined when it was first compiled to be registered to be in place available to the agent.emre
05/17/2022, 3:44 PMregister
your flow, the entire flow object is serialized with cloudpickle
(this includes tasks and edges).
• The bytecode IS NOT sent to prefect cloud, only metadata about your flow (task names, edges, storage definition and run configs etc.) is sent to prefect cloud. This prevents prefect cloud from seeing your code.
• The bytecode is sent to whatever you have configured for the flow storage.
• Your agent sees that you want to run a flow, and tries to execute it. I am presuming you are running a LocalAgent
within a docker container.
• From prefect cloud, your LocalAgent
receives not the flow bytecode, but metadata about the flow storage. The agent uses this metadata to find and fetch the bytecode itself. Then the bytecode is deserialized and executed within the python environment that you have prepared for your flow (your docker container.)Frank Hereford
05/17/2022, 3:46 PMprefect agent local start --no-hostname-label -l vision-zero -l atd-data03
)emre
05/17/2022, 3:53 PMUniversalRun
just matches with any type of Agent
, and hopes everything works out 😅 . In your case UniversalRun
simulates a LocalRun
, since your flow run is captured by a LocalAgent
Frank Hereford
05/17/2022, 3:56 PMemre
05/17/2022, 3:56 PMLocalStorage
, with stored_as_script
enabled, and path field set to whatever file your flow code was in. This works out, because your registration and execution environments are identical. Well not identical since they could have been different docker containers, but same docker image. Therefore same file layout and same pythonFrank Hereford
05/17/2022, 3:57 PMemre
05/17/2022, 3:57 PM