Channels
#prefect-server
Title
c
Casey Green
10/16/2021, 12:54 PMIs it possible to limit concurrency at the flow level without using labels?
To summarize what I'm trying to do:
1. For a specific flow, limit concurrent runs to 1.
2. No concurrency limits across flows. i.e. if I have 1000 different flows, they can all run at the same time.
3. Agents can run any flow.
In order accomplish (1), I need to use labels (according to these docs). To accomplish (2), I'd need to make a unique label per flow. To accomplish (3), I'd have to add each unique label to the agent. If I have 1000s of flows, thats quite challenging.
a
Anna Geller
10/16/2021, 2:41 PM
Hi @Casey Green, just to make sure: do you use Prefect Server or Prefect Cloud? Concurrency limits is a Cloud-only feature.
z
Zanie
10/17/2021, 5:33 PM
We don't provide limiting without tags right now. (2) is simple since you can add the name of the flow as a tag at registration time. (3) is tricky, I agree. Perhaps we can add a way to have an agent be agnostic to tags. Perhaps instead of setting the tag to the name of the flow you could just use an integer from 0-N then have all N tags on the agent. You could also create an agent subclass to retrieve all tags that exist in the backend at initialization time, this is a bit of a hack though and may not be practical at scale either.
j
Julian
10/18/2021, 8:01 AMYou can add a task at the beginning (or at any point of your flow where you think it is crucial that there is no other flow_run of this flow at the same time) of your flow execution that checks that there is no other running flow_run with your flow-id and earlier
createdc
Casey Green
10/18/2021, 2:07 PMHi @Anna Geller. We're using cloud (server in dev, but in dev this requirement is not such a big deal).
Thanks for the pointers Michael and Julian! I was afraid that the least hack-y solution here was the one you outlined Julian. Thanks again.
👍 1
3 Views