responsible for passing output of one task as input of another task in Prefect flow DAG since the Prefect server manages no code or input/output data. i'm trying to wrap my heads around what is the responsibility of the Agent. Maybe an architecture diagram might help.
z
Zanie
02/02/2021, 9:13 PM
Hi @BK Lau -- the agent is purely responsible for creating the process that runs the flow. The flow and task runners handle passing data.
b
BK Lau
02/02/2021, 9:16 PM
@Zanie Is the flow/task runners also responsible for pulling code dependencies, artifacts, configs it needs into the execution environment?
z
Zanie
02/02/2021, 9:18 PM
The process looks a bit like this:
-- Write your flow
-- Register your flow
-- Flow storage is built, flow code is written somewhere in your infrastructure
-- Flow metadata is serialized, sent to Prefect
-- Run an agent, queries Prefect for flows to run
-- Run your flow from the UI, marks the flow as ready to run
-- Agent detects flow, gets metadata from Prefect
-- Agent locates flow storage in your infra
-- Creates a process to execute the flow
Zanie
02/02/2021, 9:19 PM
That's dependent on the type of run configuration you're using. If you're using a `DockerAgent`/`DockerRun` then you just put all of that in a docker image and the agent creates a container with that image that executes your flow.
Zanie
02/02/2021, 9:19 PM
If you're using a
LocalAgent
then whatever environment your agent is in is the environment your code runs in (roughly)
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.