Yes — your mental model is mostly right, with one important correction: *This error points most strongly to a deployment concurrency lease, not work-pool / work-queue concurrency.* I dug through the Prefect 3.x orchestration behavior, and the relevant behavior is: - a deployment concurrency slot is leased for a flow run - when the flow run transitions to

PAUSED

, Prefect releases that deployment concurrency lease - when the flow resumes, Prefect does not cleanly reacquire a fresh deployment lease for that resumed execution path - instead, the client-side lease renewer can try to renew the old lease ID, which is already gone - that produces:

Concurrency lease renewal failed - slots are no longer reserved

So the issue is not just “too many resumed at once” in the abstract — it’s specifically consistent with deployment-level concurrency lease handling around pause/resume. Answers to your questions 1. Is the subset failure pattern consistent with a concurrency limit lower than 30? Partly yes, but with nuance. If only some resumed runs failed while others resumed fine, that is consistent with a concurrency-controlled resource being contended. But for this exact error message, the more precise interpretation is: - those runs had a deployment concurrency lease associated with them - that lease was released when they paused - on resume, some runs hit lease-renewal failure because their old reservation no longer existed So “they competed for slots and some lost” is directionally reasonable, but the actual mechanism is more like: - pause released the lease - resume/renewal encountered a stale or missing lease - the run then terminated because the lease could no longer be renewed That’s why this points more to deployment concurrency leases than to queue/pool throttling. 2. Where should you look for the actual limit? The first place I would check is: 1. Deployment concurrency limit for the deployment(s) involved Then also check: 2. Global concurrency limits in the workspace 3. Work pool concurrency limit 4. Work queue concurrency limit — though you already said this is unlimited But the key distinction is: - work pool / work queue concurrency control scheduling/launching of runs - the lease renewal error is tied to Prefect’s concurrency lease mechanism, which is most directly associated with deployment concurrency So if I were narrowing it down, I’d start with the deployment config before anything else. Other places to look: - whether multiple deployments share the same underlying concurrency-limited resource - whether any automation or orchestration policy is mutating state during resume - whether you’re actually on Prefect 2.x semantics in Cloud for this workload, since your earlier setup mentioned 2.x; behavior differs by version But for this exact error string, deployment concurrency is the strongest match. 3. Do paused runs keep the slot, or are the options basically “raise the limit” or “stagger resumes”? Based on the orchestration behavior I found: - paused runs do not retain the deployment concurrency slot - the lease is released on pause - there is not currently a clean “release on pause, reacquire on resume” path that re-queues the run gracefully instead of failing - so yes, operationally your options are mostly: - raise the effective concurrency limit high enough for your burst - stagger resumes so you don’t create a resume storm - avoid pause/resume on concurrency-limited deployments if possible So your statement:

for work-pool / deployment / global concurrency limits, the "release before pause, re-acquire on resume" pattern doesn't apply

needs a slight tweak: - for deployment concurrency, Prefect does release the lease on pause - but it does not reliably reacquire it cleanly on resume - which is why you see termination instead of a clean re-queue/wait

For work-pool/work-queue concurrency, those are not the ones I’d expect to produce this lease-renewal message. Practical next steps I’d check these in order: 1. Deployment settings - inspect the deployment(s) for any concurrency limit 2. Workspace global concurrency limits - especially anything that could apply indirectly to these runs 3. Work pool concurrency - worth checking, but less likely for this exact error 4. Whether all failing runs came from the same deployment - if yes, that makes deployment concurrency even more likely If you want, I can help you build a checklist of exactly where to click / which CLI commands to run to inspect: - deployment concurrency - work pool concurrency - global concurrency limits and map each one to whether it can plausibly cause this specific error. Relevant docs / references - Prefect docs: Prefect documentation - Global concurrency limits: Global concurrency limits - Deployments: Deployments - Work pools: Work pools If you want, I can also help you determine whether this is likely a Prefect bug/edge case versus expected behavior, especially if you confirm whether you’re on Prefect 2.x or 3.x for this environment.