I am wondering if there is a mechanism in prefect for flow looping/mapping. Ie. I want to run a flow with N inputs and collect those outputs for a downstream task. I could write a for loop within the flow, but this creates a very wide DAG which becomes unwieldy after 1000 iterations.

We solved a similar problem with mapped tasks and “sub” flows - a sub flow being a flow we trigger within a mapped task using the FlowRunTask. The DAG remains contained, but the tasks at runtime can explode out a little 🙂 You’d also need to implement a check inside the mapped task to poll for when the sub flow completes, but it works surprisingly well in our case

You’d also need to implement a check inside the mapped task to poll for when the sub flow completes

If you're using the

FlowRunTask

, note that there's the

wait

kwarg which if set to

True

will wait until the flow run has completed and set the task state as the result of the run flow.

Oh didn’t know about that, thanks @Jim Crist-Harif very useful!!

Thanks! I assume there is no local way of executing this yet. I look forward to subflow executions.