@Marvin When I send a deployment to a workpool that contains 100 concurrent tasks, how do I make it so that the 100 concurrent tasks are run on a distributed amount of agents, instead of all tasks on one agent?

To ensure that 100 concurrent tasks from a deployment run across multiple workers instead of all on one worker, you should configure your work pools and scaling method appropriately. Here are some key points and relevant documentation to guide you: ### Work Pools - Work pool types: Prefect supports different types of work pools, such as pull and push work pools. Pull work pools require workers to poll for flow runs, while push work pools can submit runs directly to serverless infrastructures. - Scalability: It's essential to design your system for scalability, managing multiple task queues that run tasks on specific target infrastructure and distributing them at scale. ### Scaling with Dask - DaskTaskRunner: To distribute tasks across workers when using Dask as the executor, you can specify the number of workers and threads per worker in the

DaskTaskRunner

configuration. - Example:

python
    DaskTaskRunner(cluster_kwargs={"n_workers": 4, "threads_per_worker": 2})

- Dask Collections: For distributing Dask collections across workers to achieve parallel computations, you can use

get_dask_client

or

get_async_dask_client

as context managers. ### Links to Documentation - Work Pools and Workers Overview - Scaling and Managing Multiple Task Queues - DaskTaskRunner Usage Guide To distribute your tasks across multiple workers specifically, you would need to: 1. Set up a work pool with the appropriate type that matches your execution environment and aims for scalability. 2. Configure multiple workers that poll from the work pool or are triggered by it (depending on the work pool's type). 3. Define your flow runs with a scalable task runner like

DaskTaskRunner

, customizing the

cluster_kwargs

to create multiple workers as needed. By configuring this setup, your deployed flow should distribute the 100 concurrent tasks across the available workers, effectively achieving distributed execution. If you encounter issues, it is suggested to consider monitoring, centralized logging, and observability features to diagnose and manage the tasks' distribution effectively.