hey all! question! we have 3 flows kick off at the same time, all 3 flows are slightly different and take different times to execute. once all 3 flows are finished, we want to kick off ONE flow that would process data that the other 3 produced. ideally that one flow would be allowed to run once the other 3 flows are finished

it seems like a great use case for a flow of flows in 1.0 https://discourse.prefect.io/tag/flow-of-flows or subflows in 2.0

thanks @Anna Geller and i love this blog post of yours https://www.prefect.io/guide/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/ which is VERY close to what we're doing... so taking your blog post specifically: so assume the following scenario: in 3 separate Prefect flows,

raw_customers

,

raw_orders

, and

raw_payments

are created, in parallel (lets assume that

raw_payments

took 3 hours, and the other two took minutes then, in your example, how could we make

dbt

run once all 3 have finished?

you would need to specify all those 3 tasks as

upstream_tasks=[raw_customers, raw_orders, raw_payments]

, while making sure you use the

wait_for_flow_run

task too - LMK if this is not clear and feel free to share your parent flow

upstream tasks, nice! yes... ok what would

wait_for_flow_run

wait for in this tree?

Yes exactly, you would first trigger child flows for those raw data extracts, wait for their completion with this wait_for_flow_run task and then run your final child flow running DBT or something else whatever should come next as needed