Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, about Prefect flows and agents : does 1 flow == 1 agent, or are tasks in flows parallelized over the available agents ?
Should I use Dask for that ?

Agents are responsible for managing the launching of flows. A single agent can run multiple flows depending on your setup. Dask is for task-level parallelization like when you map over inputs.

<@UQVJQAK5H> But it's not restricted to mapped-tasks only, right? It's enough that you have independent branches in your DAG to take advantage of the speedup offered by Dask?

<@U01BVQ9FGLE> that’s correct :thumbsup:

Prefect + Dask help ensure that 1) your tasks are executed as fast as infrastructure becomes available 2) your task dependencies are respected