Hello all, I'm doing an ETL with a giant flow that...
# prefect-community
a
Hello all, I'm doing an ETL with a giant flow that is made out of several subflows. I'm trying to break my main flow and keep the subflows in different modules. Is there an easy way to do this? Should I inherit the Flow object in every subflow and manually define the dependencies of tasks in each subflow or is there a way to do it simple similar to how it can be done by the
with
statement definition of a flow? Does anyone have an example?
k
Hey @Arash Roshani, Check out this Prefect Idiom on running dependent flows. If you break up your mega flow, you can use this orchestrator pattern and the StartFlowRun task with a backend API to start flows with the granular configuration of the good ol’ Prefect task and also configure each flow separately (in terms of flow level configurations).
❤️ 1
👍 1
a
Thanks a lot @Kyle Moon-Wright, I'll dig into this right now
👍 1