Hi, I’m running some flows on prefect cloud and tasks are running sequentially instead of in paralle...
t
Hi, I’m running some flows on prefect cloud and tasks are running sequentially instead of in parallel. The reason I decided to use prefect is to easily parallelize task execution. I am using
task.map
. What am I doing wrong?
k
The default executor is a sequential executor that does not parallelize. You need to set the LocalDaskExecutor.
flow.executor = LocalDaskExecutor()
Docs is here
t
Does it work also with
DaskExecutor
?
k
Yes if you have a cluster. Yes you need LocalDask or Dask to parallelize
t
when I do that and re-register the flow to the prefect-cloud I get `Skipped (metadata unchanged)`and still seeing sequential execution in the prefect cloud ui.
k
If you register using the CLI, you can force the change with
prefect register -f ..
t
thanks.