https://prefect.io logo
c

Casey Green

10/16/2021, 12:54 PM
Is it possible to limit concurrency at the flow level without using labels?
To summarize what I'm trying to do: 1. For a specific flow, limit concurrent runs to 1. 2. No concurrency limits across flows. i.e. if I have 1000 different flows, they can all run at the same time. 3. Agents can run any flow. In order accomplish (1), I need to use labels (according to these docs). To accomplish (2), I'd need to make a unique label per flow. To accomplish (3), I'd have to add each unique label to the agent. If I have 1000s of flows, thats quite challenging.
a

Anna Geller

10/16/2021, 2:41 PM
Hi @Casey Green, just to make sure: do you use Prefect Server or Prefect Cloud? Concurrency limits is a Cloud-only feature.
z

Zanie

10/17/2021, 5:33 PM
We don't provide limiting without tags right now. (2) is simple since you can add the name of the flow as a tag at registration time. (3) is tricky, I agree. Perhaps we can add a way to have an agent be agnostic to tags. Perhaps instead of setting the tag to the name of the flow you could just use an integer from 0-N then have all N tags on the agent. You could also create an agent subclass to retrieve all tags that exist in the backend at initialization time, this is a bit of a hack though and may not be practical at scale either.
j

Julian

10/18/2021, 8:01 AM
You can add a task at the beginning (or at any point of your flow where you think it is crucial that there is no other flow_run of this flow at the same time) of your flow execution that checks that there is no other running flow_run with your flow-id and earlier
created
timestamp. If that condition is not satisfied you can end the flow_run. This might not be pretty, but then you would satisfy all of your points at once
c

Casey Green

10/18/2021, 2:07 PM
Hi @Anna Geller. We're using cloud (server in dev, but in dev this requirement is not such a big deal). Thanks for the pointers Michael and Julian! I was afraid that the least hack-y solution here was the one you outlined Julian. Thanks again.
👍 1
3 Views