Hi, I’m running some flows on prefect cloud and tasks are running sequentially instead of in parallel. The reason I decided to use prefect is to easily parallelize task execution. I am using
task.map
.
What am I doing wrong?
k
Kevin Kho
01/31/2022, 5:10 PM
The default executor is a sequential executor that does not parallelize. You need to set the LocalDaskExecutor.
Yes if you have a cluster. Yes you need LocalDask or Dask to parallelize
t
Talmaj Marinč
01/31/2022, 5:28 PM
when I do that and re-register the flow to the prefect-cloud I get `Skipped (metadata unchanged)`and still seeing sequential execution in the prefect cloud ui.
k
Kevin Kho
01/31/2022, 5:29 PM
If you register using the CLI, you can force the change with
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.