Hi there, I am currently evaluating different ways to manage dependencies between pipelines and come across this great article here. I think flow of flows/ master flows is great idea but wonder two things • Does Prefect still have similar

ExternalTaskSensor

task to let a master flow wait for another master flow? Or we need to create another master flow on top of previous flows • I would imagine these master flows will expand and become more complex. Also child flows inside might be overlapped e.g one extract flow is needed in two master flows. What would you recommend regarding this? Thanks

Glad you liked the post, I love subflows! #1 Yes, 100% - there is a task called

wait_for_flow_run

which polls for a child flow run state and blocks until child flow is completed before moving on to the downstream task. Here are some examples that show this in action https://github.com/anna-geller/orchestrator-pattern #2 The beauty of subflows and the orchestrator pattern is that they are completely modular pieces of workflow logic that can be called from multiple parent flows but still all execution metadata is collected within the respective child flow itself. So you can register one (child) flow and then call it in 10 other flows (parent flows) with no duplication of code or workflow logic and btw in Prefect 2.0, this will get even easier with subflows https://orion-docs.prefect.io/concepts/flows/#subflows

Thanks @Anna Geller, yes I enjoyed really much the article and examples that you gave. I still have questions regarding master flows having overlapping subflows. One scenario could be this: Master flow M1 -> Flow A --> Flow B --> Flow C Master flow M2 -> Flow A --> Flow D --> Flow E with master flow setup, Flow A (non-parametrized e.g. like extract load process) is called in each master flow, possibly two master flows will have different schedule. How would I make sure Flow A is not ran twice and B,D probably listen to the same upstream flow. Theoretically I could put these 2 into 1 master flow, but I dont think expanding continuously would be the sustainable solution.

How would I make sure Flow A is not ran twice

I didn't understand the problem you are trying to solve - could you explain a bit more on a business level? to prevent Flow A running in both flows, you would only include it in one flow (e.g. in M1) instead of in both. and don't forget that you may trigger M2 from M1, too!

hi, thanks for the patience. I would an example as follow: • We have a big extract-load flow or flow A which is like a centre point of the spider web. • It is the prerequisite to run a lot of BI pipelines e.g. reporting, dashboard refreshment, modelling. • I would assume each of pipelines is a master-flow So I wonder if it is a good sustainable structure: • There will be a master flow M to have flow A and trigger the rest of the master flows (M1,M2…MM) which depend on flow A. Does it sound like what you suggest? • Is there a usage for

idempotent key

here that helps me not re-run flow A ? • Also wonder in your explanation, if I only include flow A in M1 and not trigger M2. M2 runs by its schedule, will it run flow A again? I hope my questions are clear. Thanks a lot

I would assume each of pipelines is a master-flow

not everything can be a master flow - without any child flows, there cannot be any parent flow

• Is there a usage for

idempotent key

here that helps me not re-run flow A ?

you may use caching for that - check this discussion which covers exactly the same topic and is trying to solve a similar problem

thanks a lot 👍

No repo available, only the code examples in this Discourse topic so far