Alireza
02/14/2023, 8:07 PMMarvin
02/14/2023, 8:08 PMAlireza
02/14/2023, 8:09 PMMarvin
02/14/2023, 8:10 PMDaskTaskRunner, you can enable adaptive scaling and specify the number of minimum and maximum workers by adding the adapt_kwargs argument in the DaskTaskRunner configuration, like this:
DaskTaskRunner(
adapt_kwargs={"minimum": 2, "maximum": 10})
Alternatively, you can use the cluster_kwargs argument to specify a fixed number of workers, as below:
DaskTaskRunner(
cluster_kwargs={"n_workers": 4})
To specify your number of workers in Cloud UI, you can make your flow a custom flow run and configure the details, including the workers parameter. For more detailed information on this feature and task runners in Prefect, you can check out the Prefect 2 Task Runners documentation [here](https://docs.prefect.io/concepts/task-runners/). I hope this helps!Alireza
02/14/2023, 8:11 PMMarvin
02/14/2023, 8:11 PMFlow.run() API and pass in any arguments needed for the flow. Once you create your custom flow run, you can configure its details, including the workers parameter. The workers parameter allows you to specify the number of workers you would like to have within your Dask cluster.
We suggest making sure your workers argument is greater than or equal to the number of tasks you have within your flow so that it can handle them all. Additionally, you may also want to consider enabling adaptive scaling, which allows the Dask cluster to scale up and down based on the current workload. To do this, you'll need to add the adapt_kwargs argument in the DaskTaskRunner configuration.
If you need more information, the Prefect 2 Task Runners documentation has complete details on how to configure and run custom flow runs [here](https://docs.prefect.io/concepts/task-runners/). Hope this clarifies things a bit more!Alireza
02/14/2023, 8:15 PMMarvin
02/14/2023, 8:15 PMworkers parameter. You can find the documentation [here](https://docs.prefect.io/concepts/task-runners/). If you have any further questions, you can always contact the Prefect team directly through Slack or by email. I'm sure they'd be more than happy to help!