Is there an architectural diagram anywhere to explain how all of the different Prefect components fit together (e.g. agents, executors etc). Trying to wrap my head around some of the terminology and concepts after coming from a heavy Airflow background.

There is this but I dont know if that’s readable immediately

Cheers, Kevin. So am I right in my understanding that we need the following key components and their purpose: 1. Prefect Cloud / Server - essentially the Airflow scheduler if I wanted to compare this to Airflow land. It monitors, schedules and executes our flows, but it does NOT manage or execute the underlying tasks. 2. Flows - essentially an Airflow DAG (although not actually a directed a directed acyclic graph, just using the term for comparing apples-to-apples) 3. Tasks - same as in Airflow. Units of work that get executed. 4. Agent(s) - a lightweight process which polls the Cloud/Server API for new flow runs which need to be executed. Agent will monitor and manage the underlying tasks in that flow (this is where I am fuzzy, so quite likely very wrong). 5. Executor(s) - underlying compute which runs individual tasks, managed by the agent. Presuming I have the above correct(ish), I think my main confusion is around why we would push the agents to say Kubernetes or Fargate. Running our tasks on an elastic service like this makes sense, especially if the number of tasks is dynamic. Is the use case for running agents on an elastic service like this in case of dynamic flow run generation (so each flow run is run on it’s own fargate task?). I think my head is too much in Airflow land, whereby if we have say a fixed number of 200 DAGs (Flows), the elastic compute is applied to the underlying tasks in those DAGs, not the DagRuns themselves. Whereas, I think Prefect is more flexible and we can also scale up the FlowRuns (DagRuns)?

Mostly right. The agent starts up the Flow with the specified configuration (Kubernetes, Docker, ECS). Each Flow gets a different container. The Flow is responsible of running tasks. Using Kubernetes gives you the containerized execution and spins up the Flow, and then removes the container when done. Dynamicism tends to be handles by the Executor (Local/Dask). DaskExecutor can scale out the number of tasks being executed across a cluster. The main difference is Prefect doesn’t host compute. You provide your own via the agent. Prefect Cloud doesnt run anything. So the agent is more of a way to fetch the work and run it on compute infrastructure that the user provides (Kubernetes cluster for example)

Right, if they are quite different, I’ll jump to the 2.0 docs, as looking to implement 2.0 in a greenfield project

let us know if you have any specific questions, @Todd de Quincey

Thanks, Anna. I think the concepts in 2.0 (or at least the docs) make a bit more sense for me. So far so good :)

check out: • https://discourse.prefect.io/t/what-are-the-components-of-prefect-2-0-architecture/909 • https://discourse.prefect.io/t/how-to-install-prefect-2-0-on-a-linux-vm/886

Wow this looks significantly simpler than 1.0. Dev team have been hard at work.