https://prefect.io logo
Title
j

juana

04/25/2023, 4:42 PM
Hi, I'm currently trying out the dask task runner. How is the number of tasks running in parallel determined? This is the decorator I used.
@flow(task_runner=DaskTaskRunner(cluster_kwargs={"processes": False}))
b

Bianca Hoch

04/25/2023, 7:11 PM
Hey Juana! When using the Dask task runner in Prefect, the number of tasks running in parallel is determined by the Dask cluster configuration. The Dask task runner can spin up a local Dask cluster on the fly or connect with a Dask environment you've set up separately. For more information on configuring and using the Dask task runner, you can refer to the Prefect documentation here. If you're using
prefect-dask
, you can specify the number of workers and the number of threads per worker. More on that here.