Sorry if this has been asked/answered before.
One thing I really miss from other orchestrators is a simple way to specify how concurrency for a flow should behave. In Airflow I believe this is controlled with using a
LatestOnlyOperator
or using
depends_on_past
. In Oozie (I know, not my favorite orchestrator), this could be achieved by setting a
concurrency
to
1
and
execution
to
FIFO
or
ONLYLAST
.
Are there any plans in Orion to provide similar functionality so users don’t have to implement custom solutions to this common pattern?
k
Kevin Kho
03/17/2022, 4:50 PM
Yes absolutely there is a plan to make this first class in Orion. It still can be somewhat achieved in current Prefect with FlowRun Concurrency
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.