https://prefect.io logo
v

Vincent

09/22/2020, 12:52 PM
I am wondering if there is a mechanism in prefect for flow looping/mapping. Ie. I want to run a flow with N inputs and collect those outputs for a downstream task. I could write a for loop within the flow, but this creates a very wide DAG which becomes unwieldy after 1000 iterations.
d

Darragh

09/22/2020, 1:30 PM
We solved a similar problem with mapped tasks and “sub” flows - a sub flow being a flow we trigger within a mapped task using the FlowRunTask. The DAG remains contained, but the tasks at runtime can explode out a little 🙂 You’d also need to implement a check inside the mapped task to poll for when the sub flow completes, but it works surprisingly well in our case
upvote 1
j

Jim Crist-Harif

09/22/2020, 1:59 PM
You’d also need to implement a check inside the mapped task to poll for when the sub flow completes
If you're using the
FlowRunTask
, note that there's the
wait
kwarg which if set to
True
will wait until the flow run has completed and set the task state as the result of the run flow.
d

Darragh

09/22/2020, 3:12 PM
Oh didn’t know about that, thanks @Jim Crist-Harif very useful!!
v

Vincent

09/22/2020, 3:17 PM
Thanks! I assume there is no local way of executing this yet. I look forward to subflow executions.