Hello Prefect folks, I have multiple data extraction tasks that are running in separate pipelines. The longes of these can take about 3 hours. I want to create a flow that waits for the last of these to finish running before my dbt tasks run in a separate pipeline. Is there a way to ensure that the extraction tasks finish before the dbt tasks begin? We are using prefect 1.0 and we are not using the dask executor at present. Mostly we need a good enough solution until we can create a long term solution to run tasks or subflows in parallel. Thanks!

If i'm understanding what you're trying to do then I think a flow of flows pattern should work pretty well here https://discourse.prefect.io/t/how-to-schedule-and-orchestrate-a-flow-of-flows-for-a-data-warehousing-etl-use-case/691

@Mason Menges Thanks for the article, it looks promising. I am wondering if you know if in a flow of flows model the child flows can run in parallel. I know that with tasks you have to have the daskexecutor in order for it to work. Thanks again!

Hey @Chris Marchetti [Datateer] generally speaking yes, provided your not waiting for the flow run the create_flow_run task is just an api call so each flow run can and will run independently 😄