https://prefect.io logo
Title
o

Ofir

08/26/2022, 3:57 PM
So:
with Flow("Data preprocess" as data_preprocess_flow:
  ...

with Flow("Visualization" as visualization_flow:
  ...
vs:
with Flow("Monolithic data pipeline") as monolithic_flow:
   res = data_preprocess(dataframe)
   res = train(res)
   ...
1
r

Ryan Peden

08/26/2022, 4:01 PM
In Prefect 2 you can you use subflows, which might be a good fit. I recommend reading this, especially the 'Subflows or tasks?' section.
o

Ofir

08/26/2022, 5:03 PM
Thanks a lot Ryan this is very helpful
Out of mere curiosity, how have people accomplished this in Prefect 1?
r

Ryan Peden

08/26/2022, 5:05 PM
I believe a flow of flows would be the Prefect 1 equivalent: https://docs-v1.prefect.io/core/idioms/flow-to-flow.html#scheduling-a-flow-of-flows
🙌 1