Billy McMonagle

06/29/2022, 6:14 PM
Hi there, I have a question about iterated task mapping...
I have a series of 3 mapped tasks which happen in sequence. However, each task is completely parallel-izable (they only depend on one input, which is passed on down the line). I'm using
with a
, but I'm noticing that each mapped task must completely finish before the next one begins. This leaves worker processes idle. Is there a way to get around this?
Typing it out, it strikes me that perhaps I could do a flow of flows, and that maybe that would solve my problem?

Kevin Kho

06/30/2022, 8:17 AM
Ah for Prefect 1, only one mapped task runs at a time. You could try flow of flows but the problem here is resource contention because it will try to spin up many LocalDaskExecutors unaware of each other