Yes — your diagnosis is basically right. What’s happening is: - your flow run is holding a concurrency lease - during the human-in-the-loop pause, that lease still needs to be renewed in the background - if Prefect can’t renew it in time, the server treats the lease as expired/revoked and releases the slot - when execution resumes, Prefect sees that the slot is no longer reserved and stops execution with:

Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.

So the message means: “this run no longer owns the concurrency slot it started with, so continuing would violate the concurrency limit.” A few important nuances: This is especially likely if the pause is long. Prefect leases are time-based and must be renewed periodically. If a run is paused long enough, or if renewal fails due to connectivity / worker / API issues while paused, the lease can be lost. Pause does not mean “hold the slot forever.” For long human approval steps, that’s generally not the right mental model. If this is deployment-level concurrency, the behavior is stricter. Prefect flow engine maintains a deployment concurrency lease for the run, and if renewal fails, execution is terminated to avoid over-allocation. Best practice for long human-in-the-loop waits Don’t hold scarce concurrency across the pause if you can avoid it. Instead, structure the flow like: 1. do the work that needs concurrency 2. release the constrained resource / exit concurrency context 3. pause for human input 4. on resume, re-acquire concurrency before continuing Conceptually:

with concurrency("my-limit"):
    do_pre_approval_work()

approval = pause_flow_run(...)

with concurrency("my-limit"):
    do_post_approval_work(approval)

That pattern is the safest because you are not reserving capacity while waiting on a human. If you must keep execution alive across a long pause, there are only imperfect options: - increase lease duration where configurable - tolerate lease-renewal failure in user-managed concurrency contexts with

raise_on_lease_renewal_failure=False

But that second option has an important tradeoff: Prefect may continue even though the slot was lost, which can overrun the intended limit. So I would not recommend that for strict resource protection. For your case — 30 runs on one ECS task, all paused and later resumed — I’d suspect one of these: - the runs were holding concurrency while paused - the lease TTL elapsed during the pause - or the ECS task / connectivity prevented lease renewal while they were paused So yes, your explanation is consistent with Prefect’s behavior. Relevant docs: - Global concurrency limits - How to apply global concurrency and rate limits - Interactive workflows / human-in-the-loop If you want, I can help you determine which concurrency mechanism you’re hitting here: - deployment concurrency - global concurrency limit - tag/task-run concurrency - worker/work-pool level behavior That distinction matters, because the mitigation is a bit different for each.

Thanks — that additional detail changes the diagnosis quite a bit. Given what you described, the most important point is: If you are not using

with concurrency(...)

, and the work queue flow-run concurrency is unlimited, the most likely source is deployment concurrency, not work-queue concurrency. And for paused flows specifically: Paused flow runs do not keep holding deployment concurrency forever. When a run leaves

Running

and enters

Paused

, Prefect releases the deployment concurrency slot. On resume, it has to acquire one again before continuing. So to your questions: 1. Is “some resumed, some failed” consistent with a lower concurrency limit than 30? Yes, very much so. That pattern is exactly what you’d expect if there is some engine-managed limit below 30 and all resumed runs tried to continue around the same time: - all 30 paused - their engine-managed slots were released while paused - all 30 resumed together - only some could reacquire slots - the rest lost their lease / could not validate it and were terminated So your competition model is directionally correct, with one refinement: It’s not that they kept an old lease during pause and then fought to renew it on resume. It’s more like: - pause released the slot - resume required reacquisition / validation - some succeeded - some didn’t, and the engine detected the lease was no longer valid That matches the “subset failed” behavior much better than a pure lease-expiry-during-pause explanation. 2. Where should you look for the actual limit? The main places to check are: Deployment concurrency limit This is the first place I’d look. In Prefect 3, deployment concurrency uses leases and is tightly tied to flow-run orchestration. If only some resumed runs continued, this is the strongest candidate. Work pool concurrency limit Yes, check this too. This is separate from queue concurrency. You already checked: - work queue flow run concurrency = unlimited But the work pool itself can still have its own concurrency cap. Global concurrency limits Check these as well, especially if you have any automation or workspace-level limit configured that applies indirectly to these runs. That said, global concurrency limits are usually associated with explicit concurrency usage in code or deployment-integrated features, so if you truly have no concurrency declarations in flow code, deployment/work-pool are more likely than a random workspace global limit. Other places worth checking A few other things can look similar operationally, though they are less likely to produce this exact lease message: - deployment-level concurrency options - worker/work-pool provisioning capacity constraints - automations that resume many runs at once, causing a burst against deployment/work-pool limits But for lease-based limits specifically, the important ones are: - deployment concurrency - possibly work pool concurrency if enforced in your setup - global concurrency if explicitly configured Since the error explicitly mentions a lease, I would prioritize: 1. deployment 2. work pool 3. global concurrency limits 3. Does “release before pause, reacquire on resume” apply to engine-managed limits? For deployment/work-pool/engine-managed orchestration concurrency, you don’t manually control that in user code — so yes, your intuition is right that the usual code pattern does not apply there. But there’s one subtle correction: Paused runs already do release the engine-managed deployment lease. So there is not currently a separate user action you need to take to “release before pause.” Prefect is already doing that as part of pausing. The real limitation is: you do not have a supported knob to tell Prefect, “on resume, just requeue cleanly until a slot is available instead of terminating if the lease is gone / unavailable.” So in practice, your options are mostly operational: a) Raise the relevant limit

If you want 30 paused runs to all resume together, the relevant concurrency limit needs to support that burst. b) Stagger resumes If the true limit is, say, 10, resume in batches of 10 or less. c) Redesign so not all paused runs resume simultaneously For example, have the human approval fan back in more gradually. Those are the most realistic mitigations. I would not currently assume there is a built-in mode where a paused run cleanly drops its engine-managed lease and then waits indefinitely in a queued/retry state on resume until capacity is available. Prefect’s orchestration is stricter than that, and if the lease/state validation fails, termination/cancellation is possible. What I’d check next, concretely 1. Deployment settings - open the deployment used by these runs - check whether a concurrency limit is set 2. Work pool settings - inspect the ECS work pool itself, not just the queue - look for any pool-level concurrency cap 3. Workspace concurrency limits - check global concurrency limits in the UI / API 4. Compare successful vs failed runs - were the successful ones resumed slightly earlier? - did they belong to different deployments? - did all failed runs come from the same deployment? That comparison can usually identify whether the bottleneck is deployment-specific. My short answer to your three questions: - Q1: Yes, a subset failing is consistent with a limit below 30 on resume. - Q2: Check deployment concurrency first, then work pool concurrency, then global concurrency limits. - Q3: Correct that you can’t manually manage engine-held leases the way you can with `with concurrency(...)`; practically, your remedies are raise the limit or stagger resumes. Paused runs already release deployment leases, but there isn’t a general “resume by requeueing until capacity is available” switch I can verify. If you want, I can help you build a diagnostic checklist for the exact UI/API fields to inspect for: - deployment concurrency - ECS work pool concurrency - global concurrency limits and help you narrow down which one is producing the lease.