Are flow objects mostly similar or fundamentally different in Prefect Core vs Prefect Orion? I’d be specifically interested in why in Prefect Core you decided to make Flow a context manager and not a decorator like it is now in Prefect Orion. Was that a more or less arbitrary choice, or is there a deeper reason why in the old API a decorator wasn’t possible or desirable? There’s the statement at https://www.prefect.io/blog/announcing-prefect-orion/ that now flows aren’t DAGs any more. But I couldn’t find an explanation what they are now. I presume the prefect.flow decorator still does some analysis and generates a DAG-like data structure that represents task dependencies? Is this change in what a flow object is related to the change context manager -> decorator? https://docs.prefect.io/core/concepts/flows.html https://orion-docs.prefect.io/api-ref/prefect/flows/

Hi @Christoph Deil! In Prefect Core, flows are static DAGs that are analyzed with a static pass at registration time, prior to execution. Therefore the critical demarcation of the

Flow

object was entering its context; once in that context, we could identify

Parameter

objects and

Task

dependencies through that graph analysis. This tells us the DAG structure and inputs that can be provided at runtime. Orion is a dynamic engine. This means that the flow structure is discovered at runtime and may modify itself in response to runtime activity. A flow may be run at any time - including from within another flow. Therefore, flows are (almost literally) functions in this paradigm - they have inputs and outputs and can be composed arbitrarily. Hence, a decorator is a much more natural abstraction. In Orion, strictly speaking, the post-hoc execution of a flow run remains a DAG (task runs execute in a strict order from a retrospective point of view) but the flow object itself does not inherit that constraint.

Thanks, @Jeremiah! I guess in Prefect Core there could have been a flow decorator like this that does what the Flow context manager does?https://gist.github.com/cdeil/b46bc16d81a5e2f609e83a4cfc03913a But it would have been weird because it had to execute the function body to create the DAG and that’s very uncommon for decorators?

Good question - thinking back, I think one of the major drivers of the context design was that we wanted

Parameters

to behave as

Tasks

, and be an explicit part of the DAG, so introducing a decorated function (with arguments) would have conflicted with that. But as with many things in Python it’s definitely possible :) For example the decorator in your gist could open a flow context, call the function, return the resulting flow object. However I think you’re also right that it would have introduced a (possibly) uncomfortable question of when the DAG introspection actually took place, especially if your decorated function took arguments. Maybe one way to sum up the design objectives: - Prefect Core sought to make the exercise of building static DAG structures as Pythonic as possible by introducing the functional task API (but no similar facility for flows, which were exclusively used as containers for tasks). However, this still involves separate “compile” and “run” steps to discover and execute the DAG. - Prefect Orion idealizes “code as workflows”, seeking to make all workflow operations as Pythonic as possible to minimize developer overhead. Flows are functions; tasks are functions; run them however you want.

Clarification question to see if my mental model is correct: In Orion, a flow is not one “static DAG” but a “dynamic DAG”? So do Tasks get submitted incrementally? Like “Run Task 1 and Task 2 in parallel, then get back to me, I make a if then else decision and submit the next Task?”

Yes that is right