https://prefect.io logo
Title
c

Christian Nuss

07/14/2022, 5:50 PM
hey all! question! we have 3 flows kick off at the same time, all 3 flows are slightly different and take different times to execute. once all 3 flows are finished, we want to kick off ONE flow that would process data that the other 3 produced. ideally that one flow would be allowed to run once the other 3 flows are finished
a

Anna Geller

07/14/2022, 6:30 PM
it seems like a great use case for a flow of flows in 1.0 https://discourse.prefect.io/tag/flow-of-flows or subflows in 2.0
c

Christian Nuss

07/14/2022, 6:42 PM
thanks @Anna Geller and i love this blog post of yours https://www.prefect.io/guide/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/ which is VERY close to what we're doing... so taking your blog post specifically: so assume the following scenario: in 3 separate Prefect flows,
raw_customers
,
raw_orders
, and
raw_payments
are created, in parallel (lets assume that
raw_payments
took 3 hours, and the other two took minutes then, in your example, how could we make
dbt
run once all 3 have finished?
a

Anna Geller

07/14/2022, 8:00 PM
you would need to specify all those 3 tasks as
upstream_tasks=[raw_customers, raw_orders, raw_payments]
, while making sure you use the
wait_for_flow_run
task too - LMK if this is not clear and feel free to share your parent flow
c

Christian Nuss

07/14/2022, 8:10 PM
upstream tasks, nice! yes... ok what would
wait_for_flow_run
wait for in this tree?
a

Anna Geller

07/14/2022, 10:32 PM
Yes exactly, you would first trigger child flows for those raw data extracts, wait for their completion with this wait_for_flow_run task and then run your final child flow running DBT or something else whatever should come next as needed