Hi, I am trying to undertand how

concurrency limits

work. 1. Say I set a limit of 10 on a particular work-queue (or a flow tag), and I get 15 concurrent requests for the flow run, what happens? Will the agent keep picking up runs until there are 10 runs in

running

state? will the 5 remaining runs remain in

scheduled

state? or will they go in

pending

state? Is there any doc that explains the state transitions for when concurrency limits are set? 2. Currently we don’t have any concurrency limit but I noticed that when we get a big burst of flow runs triggered, some runs go into a pending state and remain pending forever. Wondering if we’d run into similar issues with

concurrency limits

. 3. Is there a difference between concurrency limits on

work-queue

vs

tags

in terms of underlying implementation?

I have done quite a bit of this lately! I can at least answer question 1. If you have more submitted flow runs than your concurrency limit can handle, they'll enter a

late

state because they're past their scheduled time but have not yet started running. This causes your work queue to be marked as unhealthy, and you can see a count of the late flows from the work queues page. I like to watch the late flows count down as concurrency slots open up when running flows finish.

late is just a UI visualization for scheduled - they will stay scheduled

ah! ty for the clarification

there is a bug with the work-queues where if there are over 750 scheduled flow runs, they stop submitting in general, and require toggling the schedule to clear out

any thoughts on the relationship between flow runs waiting for their turn in the queue, the representation of those flows as late, and the effect on the work queue health indicator? I'm not sure it feels intuitive that flows that are intentionally "late" should imply that a work queue is unhealthy

I think this answer might differ between v1 and v2 - if this is V1, while I agree, I don’t think will be addressed. If this is v2, I think the intent of it was to get your attention although, agreed maybe “unhealthy” is not the proper description for it

using v2 here. It might be helpful to distinguish flows that are late because the concurrency limit is reached vs other reasons, but if there's no mechanism for determining that within prefect that's ok too