Linh Nguyen
05/18/2022, 7:38 AMExternalTaskSensor
task to let a master flow wait for another master flow? Or we need to create another master flow on top of previous flows
• I would imagine these master flows will expand and become more complex. Also child flows inside might be overlapped e.g one extract flow is needed in two master flows. What would you recommend regarding this?
ThanksAnna Geller
wait_for_flow_run
which polls for a child flow run state and blocks until child flow is completed before moving on to the downstream task. Here are some examples that show this in action https://github.com/anna-geller/orchestrator-pattern
#2 The beauty of subflows and the orchestrator pattern is that they are completely modular pieces of workflow logic that can be called from multiple parent flows but still all execution metadata is collected within the respective child flow itself. So you can register one (child) flow and then call it in 10 other flows (parent flows) with no duplication of code or workflow logic
and btw in Prefect 2.0, this will get even easier with subflows https://orion-docs.prefect.io/concepts/flows/#subflowsLinh Nguyen
05/19/2022, 8:33 AMAnna Geller
How would I make sure Flow A is not ran twiceI didn't understand the problem you are trying to solve - could you explain a bit more on a business level? to prevent Flow A running in both flows, you would only include it in one flow (e.g. in M1) instead of in both. and don't forget that you may trigger M2 from M1, too!
Linh Nguyen
05/19/2022, 2:16 PMidempotent key
here that helps me not re-run flow A ?
• Also wonder in your explanation, if I only include flow A in M1 and not trigger M2. M2 runs by its schedule, will it run flow A again?
I hope my questions are clear. Thanks a lotAnna Geller
I would assume each of pipelines is a master-flownot everything can be a master flow - without any child flows, there cannot be any parent flow
• Is there a usage foryou may use caching for that - check this discussion which covers exactly the same topic and is trying to solve a similar problemhere that helps me not re-run flow A ?idempotent key
Linh Nguyen
05/23/2022, 6:02 AMAnna Geller