Thread
#prefect-server
    Casey Green

    Casey Green

    11 months ago
    Is it possible to limit concurrency at the flow level without using labels?
    To summarize what I'm trying to do:1. For a specific flow, limit concurrent runs to 1. 2. No concurrency limits across flows. i.e. if I have 1000 different flows, they can all run at the same time. 3. Agents can run any flow. In order accomplish (1), I need to use labels (according to these docs). To accomplish (2), I'd need to make a unique label per flow. To accomplish (3), I'd have to add each unique label to the agent. If I have 1000s of flows, thats quite challenging.
    Anna Geller

    Anna Geller

    11 months ago
    Hi @Casey Green, just to make sure: do you use Prefect Server or Prefect Cloud? Concurrency limits is a Cloud-only feature.
    Michael Adkins

    Michael Adkins

    11 months ago
    We don't provide limiting without tags right now. (2) is simple since you can add the name of the flow as a tag at registration time. (3) is tricky, I agree. Perhaps we can add a way to have an agent be agnostic to tags. Perhaps instead of setting the tag to the name of the flow you could just use an integer from 0-N then have all N tags on the agent. You could also create an agent subclass to retrieve all tags that exist in the backend at initialization time, this is a bit of a hack though and may not be practical at scale either.
    j

    Julian

    11 months ago
    You can add a task at the beginning (or at any point of your flow where you think it is crucial that there is no other flow_run of this flow at the same time) of your flow execution that checks that there is no other running flow_run with your flow-id and earlier
    created
    timestamp. If that condition is not satisfied you can end the flow_run. This might not be pretty, but then you would satisfy all of your points at once
    Casey Green

    Casey Green

    11 months ago
    Hi @Anna Geller. We're using cloud (server in dev, but in dev this requirement is not such a big deal). Thanks for the pointers Michael and Julian! I was afraid that the least hack-y solution here was the one you outlined Julian. Thanks again.