https://prefect.io logo
#best-practices
Title
# best-practices
l

Linh Nguyen

05/18/2022, 7:38 AM
Hi there, I am currently evaluating different ways to manage dependencies between pipelines and come across this great article here. I think flow of flows/ master flows is great idea but wonder two things • Does Prefect still have similar
ExternalTaskSensor
task to let a master flow wait for another master flow? Or we need to create another master flow on top of previous flows • I would imagine these master flows will expand and become more complex. Also child flows inside might be overlapped e.g one extract flow is needed in two master flows. What would you recommend regarding this? Thanks
a

Anna Geller

05/18/2022, 10:51 AM
Glad you liked the post, I love subflows! #1 Yes, 100% - there is a task called
wait_for_flow_run
which polls for a child flow run state and blocks until child flow is completed before moving on to the downstream task. Here are some examples that show this in action https://github.com/anna-geller/orchestrator-pattern #2 The beauty of subflows and the orchestrator pattern is that they are completely modular pieces of workflow logic that can be called from multiple parent flows but still all execution metadata is collected within the respective child flow itself. So you can register one (child) flow and then call it in 10 other flows (parent flows) with no duplication of code or workflow logic and btw in Prefect 2.0, this will get even easier with subflows https://orion-docs.prefect.io/concepts/flows/#subflows
l

Linh Nguyen

05/19/2022, 8:33 AM
Thanks @Anna Geller, yes I enjoyed really much the article and examples that you gave. I still have questions regarding master flows having overlapping subflows. One scenario could be this: Master flow M1 -> Flow A --> Flow B --> Flow C Master flow M2 -> Flow A --> Flow D --> Flow E with master flow setup, Flow A (non-parametrized e.g. like extract load process) is called in each master flow, possibly two master flows will have different schedule. How would I make sure Flow A is not ran twice and B,D probably listen to the same upstream flow. Theoretically I could put these 2 into 1 master flow, but I dont think expanding continuously would be the sustainable solution.
a

Anna Geller

05/19/2022, 11:54 AM
How would I make sure Flow A is not ran twice
I didn't understand the problem you are trying to solve - could you explain a bit more on a business level? to prevent Flow A running in both flows, you would only include it in one flow (e.g. in M1) instead of in both. and don't forget that you may trigger M2 from M1, too!
l

Linh Nguyen

05/19/2022, 2:16 PM
hi, thanks for the patience. I would an example as follow: • We have a big extract-load flow or flow A which is like a centre point of the spider web. • It is the prerequisite to run a lot of BI pipelines e.g. reporting, dashboard refreshment, modelling. • I would assume each of pipelines is a master-flow So I wonder if it is a good sustainable structure: • There will be a master flow M to have flow A and trigger the rest of the master flows (M1,M2…MM) which depend on flow A. Does it sound like what you suggest? • Is there a usage for
idempotent key
here that helps me not re-run flow A ? • Also wonder in your explanation, if I only include flow A in M1 and not trigger M2. M2 runs by its schedule, will it run flow A again? I hope my questions are clear. Thanks a lot
a

Anna Geller

05/20/2022, 5:09 AM
I would assume each of pipelines is a master-flow
not everything can be a master flow - without any child flows, there cannot be any parent flow
• Is there a usage for
idempotent key
here that helps me not re-run flow A ?
you may use caching for that - check this discussion which covers exactly the same topic and is trying to solve a similar problem
l

Linh Nguyen

05/23/2022, 6:02 AM
thanks a lot 👍
I think caching is cool feature. I am also curious about idempotent key use case here . Is there an article/repo that you could share for this? again big thanks.
a

Anna Geller

05/23/2022, 9:47 AM
No repo available, only the code examples in this Discourse topic so far
5 Views