Hey everyone, I’m very new to Prefect and am struggling with something simple.. I’m unable to get Prefect Server to execute multiple jobs in parallel. My guess is I have something basic misconfigured, but I’m not sure what exactly. I’ve set the executor for my flow to be

LocalDaskExecutor(scheduler="threads", num_workers=5)

but it seems like I still can only serially execute my flows. Does anyone here have experience with this? Thank you so much for the help!

Just to clarify, Prefect supports two sorts of parallel execution: 1. Running multiple flows at the same time 2. Within a flow, running multiple tasks at the same time Prefect Server has no limits on running multiple flows at the same time, you just register and schedule them. For running multiple tasks at the same time within one flow, using a Dask executor like you mentioned above is necessary but not sufficient - you also need to structure your flow in a way which allows multiple tasks to execute at the same time, most probably by using mapping.

Ah yes, makes sense. In my case, I’m trying to run multiple tasks simultaneously (within one flow). My flow is structured like the attached image; although I’m not calling map, I would think the LocalDaskExecutor would run these in parallel? Should I instead explicitly use map to get the parallelization? Thanks a lot for the help with this Greg!

Yes I would definitely use task mapping in this situation to get parallelisation. It might be worth running

flow.visualize()

to see the generated DAG which Prefect will use to execute the run, this will show how Prefect associates the tasks with each other and which dependencies each task would need to wait for in order to run.

Thank you, this is very helpful! Looking at flow.visualize(), it seems like all the dependencies are met for my nodes. I’ve also tried this with another workflow that uses map, and it seems like map is also executing one at a time. Right before I register my flow, I set

flow.executor = LocalDaskExecutor(scheduler='threads')
flow.run_config = LocalRun()

should I be setting another variable on my flow before registering? Or is there perhaps something I need to run at the agent level to make it so I can run a local dask executor? Thanks again for the help!

Glad to have helped!