Kevin Mullins03/24/2022, 7:47 PM
with 32 threads. I’ve verified in logs and the schematic that the
task should be ready to go as all it’s upstream tasks are complete; however, it still waits around for the unrelated
task to complete before starting. Was hoping someone might have an idea how to figure out what’s going on. I’ve created another example flow that had a bunch of unrelated root/child tasks and verified it seems to behave as I would expected, but haven’t tried to reproduce this with a test flow that uses mapping yet. I’ll attach a screenshot that visually shows (hopefully) what I’m talking about.
Kevin Kho03/24/2022, 8:08 PM
for the LocalDaskExecutor if you’re open to that. It tends to be more stable
Kevin Mullins03/24/2022, 8:10 PM
Kevin Kho03/24/2022, 8:21 PM
Kevin Mullins03/24/2022, 8:23 PM
Kevin Kho03/24/2022, 8:25 PM
The way prefect is currently written, we guarantee that tasks in a *single map task can run in parallel, but multiple unrelated tasks (or map tasks as above) may or may not depending on the internals to the FlowRunner implementation.
Kevin Mullins03/24/2022, 8:31 PM
Kevin Kho03/24/2022, 8:33 PM
Kevin Mullins03/24/2022, 8:36 PM
spin up a local dask cluster works but not
. Are there any concerns with using
in this manner?
Kevin Kho03/24/2022, 8:41 PM
Kevin Mullins03/24/2022, 8:43 PM