https://prefect.io logo
Title
a

Alireza

02/14/2023, 8:07 PM
@Marvin Is it possible to pass the number of DaskTaskrunner workers as a parameter of a flow?
m

Marvin

02/14/2023, 8:08 PM
Yes, passing the number of DaskTaskrunner workers as a parameter of a flow is possible. Prefect with DaskTaskrunner creates a local Dask cluster to run multiple tasks in parallel. It's important to note that you'll need to install the prefect-dask collection and use the .submit method when calling functions, which will create a task run for each task and track the status of each task. You can find more detailed information about this in the Prefect 2 Task Runners documentation [here](https://docs.prefect.io/concepts/task-runners/).
a

Alireza

02/14/2023, 8:09 PM
@Marvin show me how to pass the number of Dasktaskrunner workers as parameter to a flow in Cloud UI
m

Marvin

02/14/2023, 8:10 PM
Ah, I see. You want to pass the number of DaskTaskrunner workers as a parameter of a flow in Cloud UI. No problem, this can easily be done. When using the Prefect
DaskTaskRunner
, you can enable adaptive scaling and specify the number of minimum and maximum workers by adding the
adapt_kwargs
argument in the
DaskTaskRunner
configuration, like this:
DaskTaskRunner(
 adapt_kwargs={"minimum": 2, "maximum": 10})
Alternatively, you can use the
cluster_kwargs
argument to specify a fixed number of workers, as below:
DaskTaskRunner(
 cluster_kwargs={"n_workers": 4})
To specify your number of workers in Cloud UI, you can make your flow a custom flow run and configure the details, including the
workers
parameter. For more detailed information on this feature and task runners in Prefect, you can check out the Prefect 2 Task Runners documentation [here](https://docs.prefect.io/concepts/task-runners/). I hope this helps!
a

Alireza

02/14/2023, 8:11 PM
@Marvin Can you explain more about making my flow as custom flow run and configuring the details?
m

Marvin

02/14/2023, 8:11 PM
It sounds like you'd like to know a bit more about making your flow a custom flow run and configuring the details. I'm happy to provide more information on this. Creating a custom flow run requires you to use the
Flow.run()
API and pass in any arguments needed for the flow. Once you create your custom flow run, you can configure its details, including the
workers
parameter. The
workers
parameter allows you to specify the number of workers you would like to have within your Dask cluster. We suggest making sure your
workers
argument is greater than or equal to the number of tasks you have within your flow so that it can handle them all. Additionally, you may also want to consider enabling adaptive scaling, which allows the Dask cluster to scale up and down based on the current workload. To do this, you'll need to add the
adapt_kwargs
argument in the
DaskTaskRunner
configuration. If you need more information, the Prefect 2 Task Runners documentation has complete details on how to configure and run custom flow runs [here](https://docs.prefect.io/concepts/task-runners/). Hope this clarifies things a bit more!
a

Alireza

02/14/2023, 8:15 PM
@Marvin can you provide a link to a documentation or code sample for custom flow run?
m

Marvin

02/14/2023, 8:15 PM
Ah, I understand you would like to see a link to a documentation or code sample for custom flow run. Certainly! Prefect has several resources available to help you out with custom flow runs in the Prefect 2 Task Runners documentation. There, you can find an overview of custom flow runs, as well as several code samples which show how to create and configure them, including how to set the
workers
parameter. You can find the documentation [here](https://docs.prefect.io/concepts/task-runners/). If you have any further questions, you can always contact the Prefect team directly through Slack or by email. I'm sure they'd be more than happy to help!