Kevin Mullins
03/24/2022, 7:47 PMLocalDaskExecutor
with 32 threads. I’ve verified in logs and the schematic that the Discover
task should be ready to go as all it’s upstream tasks are complete; however, it still waits around for the unrelated Sync
task to complete before starting.
Was hoping someone might have an idea how to figure out what’s going on. I’ve created another example flow that had a bunch of unrelated root/child tasks and verified it seems to behave as I would expected, but haven’t tried to reproduce this with a test flow that uses mapping yet. I’ll attach a screenshot that visually shows (hopefully) what I’m talking about.Kevin Kho
03/24/2022, 8:08 PMprocesses
for the LocalDaskExecutor if you’re open to that. It tends to be more stableKevin Mullins
03/24/2022, 8:10 PMKevin Kho
03/24/2022, 8:21 PMKevin Mullins
03/24/2022, 8:23 PMKevin Kho
03/24/2022, 8:25 PMThe way prefect is currently written, we guarantee that tasks in a *single map task can run in parallel, but multiple unrelated tasks (or map tasks as above) may or may not depending on the internals to the FlowRunner implementation.
Kevin Mullins
03/24/2022, 8:31 PMKevin Kho
03/24/2022, 8:33 PMKevin Mullins
03/24/2022, 8:36 PMDaskExecutor
spin up a local dask cluster works but not LocalDaskExecutor
. Are there any concerns with using DaskExecutor
in this manner?Kevin Kho
03/24/2022, 8:41 PMKevin Mullins
03/24/2022, 8:43 PM