Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi there, I have a question about iterated task mapping...

I have a series of 3 mapped tasks which happen in sequence. However, each task is completely parallel-izable (they only depend on one input, which is passed on down the line). I'm using `KubernetesRun` with a `LocalDaskExecutor` , but I'm noticing that each mapped task must completely finish before the next one begins. This leaves worker processes idle. Is there a way to get around this?

Typing it out, it strikes me that perhaps I could do a flow of flows, and that maybe that would solve my problem?

Ah for Prefect 1, only one mapped task runs at a time. You could try flow of flows but the problem here is resource contention because it will try to spin up many LocalDaskExecutors unaware of each other