Thread
#prefect-community
    Chu

    Chu

    2 months ago
    Hi community, if we do not use flow of flows framework, is there any downside? Basically from my understanding flow of flows is to use one single flow (orchestrator) to manage other flows (dependency, schedule etc.) Instead of defining those relationship in each flow's script?
    Sylvain Hazard

    Sylvain Hazard

    2 months ago
    Hi ! Your understanding is very correct. It is a way to orchestrate multiple flows that need to run in order. Another common use case is to separate parts of a big flow so that they can run on their own if needed e.g. an end to end machine learning flow that orchestrates a data processing flow, a feature engineering, a training flow, etc. If you don't need to do this, you really don't need to and it actually makes code slightly more readable in my opinion.
    Anna Geller

    Anna Geller

    2 months ago
    Chu, it depends on your use case - flow of flows is particularly helpful for larger data warehousing projects where you may run hundreds of jobs to populate and transform your data warehouse tables and doing it all in a single flow without something like flow-of-flows may end up like this (which is nightmare to maintain)
    or even worse 😂
    Sylvain Hazard

    Sylvain Hazard

    2 months ago
    I expect I'll have nightmares from this tonight. Thanks Anna 😛
    Anna Geller

    Anna Geller

    2 months ago
    you're very welcome 😄
    Directed Acyclic Ghoul
    Sylvain Hazard

    Sylvain Hazard

    2 months ago
    Chu

    Chu

    2 months ago
    Thanks for your vivid explanation, successfully sell me the idea 😉 . One last question, if going with the nightmare like that, we need to define flow dependency when we created each flow, right? Like I already defined 5 flows, they have some dependencies. In flow of flows, I can just create a high level flow to mange those 5 flows. If nor using flow of flow, do I need to set those dependency in each of flows? If not, how can i do that?
    Sylvain Hazard

    Sylvain Hazard

    2 months ago
    Maybe we misunderstood your initial question. If your flows have some dependencies between them like the order in which they need to run, I'd say flow of flows is the best approach in order to manage it properly.
    Depending on the kind of dependency, you could also have a flow start the next one before finishing e.g.
    with Flow() as f:
       task_a()
       task_b()
       StartFlowRun("flow_b")
    Without knowing the full detail, I wouldn't advise doing this because it splits the "high level workflow" between each flow and it makes maintenance a pain on the long run. For example if you have something like flow A => flow B => flow C and need to change it to flow A => flow C => flow B, then you'd have to change all flows in your code.