Alright, not sure what kind of monster I’ve created. I’m created a flow that two distinct paths that are unrelated and don’t share any dependencies other than a few secrets. For some reason one of the mapped tasks won’t start until an unrelated task finishes. I’m using

LocalDaskExecutor

with 32 threads. I’ve verified in logs and the schematic that the

Discover

task should be ready to go as all it’s upstream tasks are complete; however, it still waits around for the unrelated

Sync

task to complete before starting. Was hoping someone might have an idea how to figure out what’s going on. I’ve created another example flow that had a bunch of unrelated root/child tasks and verified it seems to behave as I would expected, but haven’t tried to reproduce this with a test flow that uses mapping yet. I’ll attach a screenshot that visually shows (hopefully) what I’m talking about.

That sounds like you could just be running out of threads? Not sure. You can try

processes

for the LocalDaskExecutor if you’re open to that. It tends to be more stable

I just gave it 32 to make sure it wasn’t starved. At the time it is waiting there is just a single task running.

Does a similar pattern work in another setup? Reading again, I think this is because of a Prefect 1.0 limitation in that only 1 mapped task can run at a time even if you have available cores

The one it is waiting for is a mapped task. The one that it waits on has no mapping along it’s branch from the root task to what is running, so I wouldn’t think it would conflict.

Let me rephrase, the Flow Runner naively runs one “type of task” at a time (inefficiently). Let me dig the issue for this

Yea, sounds like you might be correct there.

I think the FlowRunner is different ground up though in 2.0, so I think this issue should not persist

Interesting that letting

DaskExecutor

spin up a local dask cluster works but not

LocalDaskExecutor

. Are there any concerns with using

DaskExecutor

in this manner?

Not at all and we do recommend that for even more stability than the processes

Awesome, problem solved then 🙂. I’ll start using it instead, thanks!