Hi, about Prefect flows and agents : does 1 flow == 1 agent, or are tasks in flows parallelized over the available agents ?
Should I use Dask for that ?
j
Jacob Blanco
10/12/2020, 12:47 AM
Agents are responsible for managing the launching of flows. A single agent can run multiple flows depending on your setup. Dask is for task-level parallelization like when you map over inputs.
👍 1
upvote 2
k
Krzysztof Nawara
10/12/2020, 7:36 AM
@Jacob Blanco But it's not restricted to mapped-tasks only, right? It's enough that you have independent branches in your DAG to take advantage of the speedup offered by Dask?
d
Dylan
10/12/2020, 3:23 PM
@Krzysztof Nawara that’s correct 👍
Prefect + Dask help ensure that 1) your tasks are executed as fast as infrastructure becomes available 2) your task dependencies are respected