https://prefect.io logo
Title
t

Tom

03/08/2022, 1:06 PM
Hi all, we are evaluating Prefect and were wondering if it is a good idea to initiate flows within flows
b

Brian Phillips

03/08/2022, 1:09 PM
Sure, we use that pattern often. One example is a "syncer" flow which tracks a source of files. Any time there are new files to ingest it kicks off a separate "copy" flow to ingest that set of files.
s

Sylvain Hazard

03/08/2022, 1:09 PM
Hello and welcome ! 👋 As far as I know, it is a common pattern that is even pretty well documented : flow of flow. The usual pattern is having atomic flows that you sometimes want to run by themselves and a "meta" flow that runs those flows in order.
t

Tom

03/08/2022, 1:10 PM
Ah nice, thanks both, I think that answers our question 🙂
s

Sylvain Hazard

03/08/2022, 1:12 PM
Also I believe Orion will allow you to run subflows in the same way you run tasks, making this pattern quite a bit easier to setup and debug.
:upvote: 4
t

Tom

03/08/2022, 1:44 PM
I haven't come across Orion yet in the docs
https://www.prefect.io/orion/ ah this one, let's have a look 🙂
s

Sylvain Hazard

03/08/2022, 1:49 PM
Be careful, it is still offered as a technical preview and is not yet deemed production ready by the Prefect team. From what was announced, we can expect to get out of preview in the coming months but nothing more precise as of yet. Depending on how soon you need to use it, it could be a good idea to stick to Prefect.
t

Tom

03/08/2022, 1:51 PM
we are in the evaluation phase now, so good to have a look where things are going anyway
s

Sylvain Hazard

03/08/2022, 1:53 PM
Definitely agree with this ! Just wanted to make sure you wouldn't go full production in the next weeks.
t

Tom

03/08/2022, 1:54 PM
hehe yeah I think the Prefect team will be more quick than our product 😉 thanks for the headsup anyway!
a

Anna Geller

03/08/2022, 2:04 PM
@Sylvain Hazard and @Brian Phillips explained it so well 💯 but if you want to read more about the flow of flows orchestration pattern, we have an entire category in Discourse devoted to that - check this out here: https://discourse.prefect.io/tag/flow-of-flows
👀 1
j

Jeremiah

03/08/2022, 2:05 PM
Plan is for Orion to graduate from tech preview to beta this month!
:party-parrot: 10
:upvote: 14