https://prefect.io logo
Title
n

Nikhil Jain

02/01/2023, 8:34 PM
Hi, I am trying to undertand how
concurrency limits
work. 1. Say I set a limit of 10 on a particular work-queue (or a flow tag), and I get 15 concurrent requests for the flow run, what happens? Will the agent keep picking up runs until there are 10 runs in
running
state? will the 5 remaining runs remain in
scheduled
state? or will they go in
pending
state? Is there any doc that explains the state transitions for when concurrency limits are set? 2. Currently we don’t have any concurrency limit but I noticed that when we get a big burst of flow runs triggered, some runs go into a pending state and remain pending forever. Wondering if we’d run into similar issues with
concurrency limits
. 3. Is there a difference between concurrency limits on
work-queue
vs
tags
in terms of underlying implementation?
❤️ 1
k

Kevin Grismore

02/01/2023, 8:38 PM
I have done quite a bit of this lately! I can at least answer question 1. If you have more submitted flow runs than your concurrency limit can handle, they'll enter a
late
state because they're past their scheduled time but have not yet started running. This causes your work queue to be marked as unhealthy, and you can see a count of the late flows from the work queues page. I like to watch the late flows count down as concurrency slots open up when running flows finish.
c

Christopher Boyd

02/02/2023, 2:25 PM
late is just a UI visualization for scheduled - they will stay scheduled
late specifically is > 30s past the scheduled start time
k

Kevin Grismore

02/02/2023, 2:26 PM
ah! ty for the clarification
c

Christopher Boyd

02/02/2023, 2:26 PM
there is a bug with the work-queues where if there are over 750 scheduled flow runs, they stop submitting in general, and require toggling the schedule to clear out
Let me try to get a good answer for 3
is this Prefect 1 or Prefect 2?
k

Kevin Grismore

02/02/2023, 2:28 PM
any thoughts on the relationship between flow runs waiting for their turn in the queue, the representation of those flows as late, and the effect on the work queue health indicator? I'm not sure it feels intuitive that flows that are intentionally "late" should imply that a work queue is unhealthy
c

Christopher Boyd

02/02/2023, 2:30 PM
I think this answer might differ between v1 and v2 - if this is V1, while I agree, I don’t think will be addressed. If this is v2, I think the intent of it was to get your attention although, agreed maybe “unhealthy” is not the proper description for it
but in some cases flows are late that will never run, while in others they are late, but just waiting for concurrency limits to free up
k

Kevin Grismore

02/02/2023, 2:32 PM
using v2 here. It might be helpful to distinguish flows that are late because the concurrency limit is reached vs other reasons, but if there's no mechanism for determining that within prefect that's ok too