Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

curious / hopefully quick question: Is it possible to combine flows into a "master flow"? Or several tasks into a "master task"? Asking here because I looked at examples, didn't see anything obvious, so figured I might be getting into anti-pattern territory. My use case is:
•  I have several scripts that can run independently. Each define a `Flow` and a `___main___` that calls `flow.run()`. 
• These scripts need to get orchestrated. I figured I'd make a master flow that knows how to execute the smaller flows and get that master flow scheduled. 

In other words: I have `script1.py`, `script2.py`, and `script3.py`. Each contain a single instance of a `Flow`. Each can be run independently of the others for various reasons.

I'd like to be able to make a `master.py` that can `import script1, script2, script3` and make a `Flow(script1.flow, script2.flow, script3.flow`

This master flow would then get scheduled. would handle failures between the smaller jobs, etc.

Hey <@UL1NMCM2R>, historically we HAVE considered this an anti-pattern, informed greatly by the negative experience of working with SubDAGs in Airflow (I believe it reached such a point that they are being removed in Airflow 2.0). While you can absolutely have a Prefect flow run another flow (it’s just a function, after all), it introduces complexity around nested behaviors that we generally discourage. With that said, over the past few months we’ve collected a number of use cases from the community that we think motivate a clean way to support “sub flows” and “flow-to-flow dependencies”. We’ve created a roadmap item to present a solid first-class way to handle it, but it is not our highest priority at this time.

As an alternative, we’d encourage you to write a function that composes your current sub flows into one large flow, if possible. You could run that function whenever you needed to update the composed function (perhaps every time!) and retain all of the benefits Prefect offers whole avoiding nested flows. 

makes sense. Thanks <@UKTUC906M>! My team and I really appreciate the guidance!