It's me again 🙃 wanted to confirm I understood something about the architecture - i'm trying to figure out the difference between : • a

Run configuration

(Local, Universal, Docker, Kubernetes) • an

Agent

(e.g. Local, Kubernetes, Docker) • and an

Executor

- (Local, LocalDask, and DaskExecutor) If i understand correctly - • our

Agent

is the entity (an always-running program, basically) which communicates with the Prefect server (and its scheduler, etc.) and brings jobs to execution phase. If it lives in Kubernetes then by default it executes flows as Kubernetes jobs , but - if we were to pick a "local"

Run configuration

- it would run it on the agent itself? • the

Run Configuration

determines where the

Agent

runs the flow (e.g., if we picked a

Docker

run_configuration for our

Docker

agent - then it would run as a docker image inside the docker image? or alongside it - as a "sibling" docker instance?) • the

Executor

determines how the flow should be executed, so for example if we had a

LocalDask

executor running in a

Docker

run_config with a

Docker

Agent - it would spin up a local Dask cluster inside the docker that's running the job? but if we picked a

DaskExecutor

then the flow would actually be executed outside the docker that's running the flow? (assuming our dask cluster runs alongside our Docker Daemon, e.g. on EC2) did i get it right or am i missing something? 😄

This is basically right!

phew

• Local runs occur in a subprocess of the agent process • Docker agents running in a docker container will run into problems unless you’ve exposed docker-in-docker. Generally, you’ll just have a docker agent process communicating with a docker engine and it will spawn a container alongside the agent. • You could specify a

DaskExecutor

with no arguments and it’d be inside the container. You could also set up a

DaskExectutor

that connects to an existing cluster outside your container or spins up a new cluster outside your container.

hmm i see, but then i guess that some setups don't make sense right? e.g. if i'm delegating task execution to a remote Dask cluster anyway, would i really care if my DAG/Flow runs in a dedicated kube job or locally in the agent?

It’s more about if you already have a cluster available I think but the kube job can take in a container with dependencies too. But yes compute will end up on the Dask cluster anyway

Yeah it might make less sense then. However, the flow orchestrator manages all of the tasks so you can get high memory consumption on the flow run’s job as it collects all the task run states.

by run state you mean data too? e.g. the data that gets passed between one task to the next?

Yeah

right.. 🤔 makes sense

Basically, yeah. Tasks aren’t intended to be that heavy. You can use subflows for that use-case.

ya i don't plan to do that, i'm just trying to stretch the possibilities to the limit to make sure i understand what's going on 😁

It would be this

not bad