flavienbwk

    flavienbwk

    1 year ago
    Hi, about Prefect flows and agents : does 1 flow == 1 agent, or are tasks in flows parallelized over the available agents ? Should I use Dask for that ?
    Jacob Blanco

    Jacob Blanco

    1 year ago
    Agents are responsible for managing the launching of flows. A single agent can run multiple flows depending on your setup. Dask is for task-level parallelization like when you map over inputs.
    Krzysztof Nawara

    Krzysztof Nawara

    1 year ago
    @Jacob Blanco But it's not restricted to mapped-tasks only, right? It's enough that you have independent branches in your DAG to take advantage of the speedup offered by Dask?
    Dylan

    Dylan

    1 year ago
    @Krzysztof Nawara that’s correct 👍
    Prefect + Dask help ensure that 1) your tasks are executed as fast as infrastructure becomes available 2) your task dependencies are respected