Hi, quick question on Orion. We are on Prefect Ser...
# ask-communityv
Vipul
12/21/2021, 6:45 PMHi, quick question on Orion. We are on Prefect Server and doing initial PoC on the Orion and would like to know if there is a way to limit the number of flow or task to one based on the parameter or context provided during the run. Our flow are very compute intensive and run overnight and we want to avoid multiple flow or task to run at the same time if they have same parameter or context as input.
z
Zanie
12/21/2021, 6:52 PM
Hi Vipul! We haven’t added any concurrency limiting to Orion yet. This feature is on the roadmap for the orchestration API, although I’m not sure it will work by comparing parameters — that’s an interesting idea to consider.
v
Vipul
12/21/2021, 7:26 PMThanks @Zanie - Is there a way we can implement by adding decorator on top of task or flow to avoid multiple task or flow to run at the same time by taking a lock in the redis
Vipul
12/21/2021, 7:29 PMAnd if there is simple snippet you could share to decorate task or flow would help us completing our PoC
z
Zanie
12/21/2021, 7:30 PM
You could do that, yes. It’d need to be written as an async function. You could use the
OrionClientFailedv
Vipul
12/21/2021, 7:31 PMThanks for the answer and I am slightly new to Prefect Orion ecosystem and so would appreciate if you can give me some snippet.
Vipul
12/21/2021, 7:31 PMThanks in advance
z
Zanie
12/21/2021, 7:32 PM
Sorry it would only have to be async if your flow is async actually/
Zanie
12/21/2021, 7:32 PM
I can’t provide a full example, I’ll write a quick snippet
Zanie
12/21/2021, 7:33 PM
Copy code
from prefect import flow
from functools import wraps
def lock_flow(fn):
@flow
@wraps(fn)
def wrapper(*args, **kwargs):
# take lock
return fn(*args, **kwargs)
return wrapper
@lock_flow
def foo(x):
return x
assert foo(2).result() == 2v
Vipul
12/21/2021, 7:34 PMWonderful @Zanie really appreciate quick response
z
Zanie
12/21/2021, 7:34 PM
Sorry, bad example 😄
v
Vipul
12/21/2021, 7:34 PMoh
z
Zanie
12/21/2021, 7:34 PM
Updated to be correct
v
Vipul
12/21/2021, 7:35 PMnice, superb
5 Views