https://prefect.io
Join Slack
Hi, quick question on Orion. We are on Prefect Server and doing initial PoC on the Orion and would l...
# ask-community
v
Hi, quick question on Orion. We are on Prefect Server and doing initial PoC on the Orion and would like to know if there is a way to limit the number of flow or task to one based on the parameter or context provided during the run. Our flow are very compute intensive and run overnight and we want to avoid multiple flow or task to run at the same time if they have same parameter or context as input.
z
Hi Vipul! We haven’t added any concurrency limiting to Orion yet. This feature is on the roadmap for the orchestration API, although I’m not sure it will work by comparing parameters — that’s an interesting idea to consider.
v
Thanks @Zanie - Is there a way we can implement by adding decorator on top of task or flow to avoid multiple task or flow to run at the same time by taking a lock in the redis
And if there is simple snippet you could share to decorate task or flow would help us completing our PoC
z
You could do that, yes. It’d need to be written as an async function. You could use the 
OrionClient
to set the state of the flow or task run to 
Failed
with a message that one is already running or you can just have it wait.
v
Thanks for the answer and I am slightly new to Prefect Orion ecosystem and so would appreciate if you can give me some snippet.
Thanks in advance
z
Sorry it would only have to be async if your flow is async actually/
I can’t provide a full example, I’ll write a quick snippet
Copy code
from prefect import flow
from functools import wraps


def lock_flow(fn):
    @flow
    @wraps(fn)
    def wrapper(*args, **kwargs):
        # take lock
        return fn(*args, **kwargs)

    return wrapper


@lock_flow
def foo(x):
    return x


assert foo(2).result() == 2
v
Wonderful @Zanie really appreciate quick response
z
Sorry, bad example 😄
v
oh
z
Updated to be correct
v
nice, superb
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by