Alireza
02/14/2023, 8:07 PMMarvin
02/14/2023, 8:08 PMAlireza
02/14/2023, 8:09 PMMarvin
02/14/2023, 8:10 PMDaskTaskRunner
, you can enable adaptive scaling and specify the number of minimum and maximum workers by adding the adapt_kwargs
argument in the DaskTaskRunner
configuration, like this:
DaskTaskRunner(
adapt_kwargs={"minimum": 2, "maximum": 10})
Alternatively, you can use the cluster_kwargs
argument to specify a fixed number of workers, as below:
DaskTaskRunner(
cluster_kwargs={"n_workers": 4})
To specify your number of workers in Cloud UI, you can make your flow a custom flow run and configure the details, including the workers
parameter. For more detailed information on this feature and task runners in Prefect, you can check out the Prefect 2 Task Runners documentation [here](https://docs.prefect.io/concepts/task-runners/). I hope this helps!Alireza
02/14/2023, 8:11 PMMarvin
02/14/2023, 8:11 PMFlow.run()
API and pass in any arguments needed for the flow. Once you create your custom flow run, you can configure its details, including the workers
parameter. The workers
parameter allows you to specify the number of workers you would like to have within your Dask cluster.
We suggest making sure your workers
argument is greater than or equal to the number of tasks you have within your flow so that it can handle them all. Additionally, you may also want to consider enabling adaptive scaling, which allows the Dask cluster to scale up and down based on the current workload. To do this, you'll need to add the adapt_kwargs
argument in the DaskTaskRunner
configuration.
If you need more information, the Prefect 2 Task Runners documentation has complete details on how to configure and run custom flow runs [here](https://docs.prefect.io/concepts/task-runners/). Hope this clarifies things a bit more!Alireza
02/14/2023, 8:15 PMMarvin
02/14/2023, 8:15 PMworkers
parameter. You can find the documentation [here](https://docs.prefect.io/concepts/task-runners/). If you have any further questions, you can always contact the Prefect team directly through Slack or by email. I'm sure they'd be more than happy to help!