https://prefect.io logo
t

Talmaj Marinč

01/31/2022, 5:10 PM
Hi, I’m running some flows on prefect cloud and tasks are running sequentially instead of in parallel. The reason I decided to use prefect is to easily parallelize task execution. I am using
task.map
. What am I doing wrong?
k

Kevin Kho

01/31/2022, 5:10 PM
The default executor is a sequential executor that does not parallelize. You need to set the LocalDaskExecutor.
flow.executor = LocalDaskExecutor()
Docs is here
t

Talmaj Marinč

01/31/2022, 5:11 PM
Does it work also with
DaskExecutor
?
k

Kevin Kho

01/31/2022, 5:12 PM
Yes if you have a cluster. Yes you need LocalDask or Dask to parallelize
t

Talmaj Marinč

01/31/2022, 5:28 PM
when I do that and re-register the flow to the prefect-cloud I get `Skipped (metadata unchanged)`and still seeing sequential execution in the prefect cloud ui.
k

Kevin Kho

01/31/2022, 5:29 PM
If you register using the CLI, you can force the change with
prefect register -f ..
t

Talmaj Marinč

01/31/2022, 5:50 PM
thanks.
2 Views