What you're seeing is expected behavior in Prefect 3.x. Why it happens A flow's deployment concurrency slot is acquired when the run enters

PENDING

, but the lease renewal loop only starts once the flow is actually running. While the run is still pending—waiting for a worker / infra / node with enough memory—there is nothing actively renewing that lease. So the lifecycle is roughly: 1. Flow run enters

PENDING

2. Concurrency slot is leased for a fixed grace period 3. If no worker/infrastructure can start it before that lease expires, Prefect considers the slot abandoned 4. The lease is repossessed and the run may be cancelled or fail to acquire concurrency when it finally tries to start This is basically a safeguard so a

PENDING

run doesn't hold a concurrency slot forever. Why Prefect does not renew in

PENDING

Because

PENDING

means "not executing yet"—often still waiting on infrastructure. The client-side renewal mechanism lives with the running flow process, so before that process exists, there's nothing to keep the lease alive. Can you avoid it without increasing grace period or adding resources? Honestly: not really, not in a direct way. If the underlying issue is "the run cannot be placed anywhere because no node has enough memory", then from Prefect's perspective the run is stuck waiting for execution capacity. Without: - a longer lease window, or - more/faster-available capacity, there is no mechanism to keep that slot indefinitely during

PENDING

. What you can do instead A few patterns may help depending on your setup: 1. Reduce startup latency - Pre-warm workers / execution environment - Use lighter images - Reduce infrastructure provisioning time 2. Avoid tying deployment concurrency to infrastructure scarcity - If this concurrency limit is being used to model business/process throttling, it's fine - But if the real bottleneck is cluster memory, deployment concurrency may be the wrong control layer 3. Lower memory requirements of the flow run - Split the flow into smaller tasks/subflows - Move heavy work to task runners / external systems - Reduce per-run resource requests if they're overestimated 4. Use work-pool / infrastructure-level scheduling controls - In Kubernetes, ECS, etc., the platform's own resource scheduling may be a better fit for this than deployment concurrency alone 5. Rely on reacquisition at start if available - In newer Prefect 3.x behavior, when a flow transitions to

RUNNING

, Prefect checks whether the lease still exists and may re-acquire it if a slot is available - But this does not solve the "stuck pending forever with no resources" case Bottom line If a run sits in

PENDING

longer than the lease because no valid node can run it, there is no built-in way to keep renewing that lease during pending. So if you want to avoid cancellation without increasing the grace period or adding capacity, your practical options are to: - make the run start faster, or - make it require fewer resources, or - change where/how you enforce concurrency If you want, I can help you think through the best pattern for your specific setup (Kubernetes worker, ECS worker, process worker, etc.) and suggest the least painful workaround.

It can, to a point — and in current Prefect 3.x that's already part of the design. When a flow goes from

PENDING

to

RUNNING

, Prefect validates the deployment concurrency lease. If the original lease expired while the run was pending, Prefect attempts to re-acquire a slot at that moment before allowing the run to proceed. So the intended behavior is: 1. run gets lease in

PENDING

2. lease expires while waiting on infra 3. when the worker finally tries to start it, Prefect checks the lease 4. if the lease is gone, Prefect tries to acquire a fresh slot 5. if that succeeds, the run continues 6. if it fails, the run is cancelled / rejected So why do you still get an error? Because reacquisition is not guaranteed. If the lease expired and, in the meantime, all concurrency slots are now occupied by other runs, Prefect must deny the start to preserve the concurrency limit. If it didn't error in that situation, Prefect would risk exceeding the configured limit. Why not silently reacquire every time? Because there are two distinct cases: Case A: slot available again - reacquire succeeds - run starts Case B: slot no longer available - reacquire fails - run cannot legally start The error is basically Prefect saying: "you were pending too long; your reservation expired; when you finally became runnable, capacity was no longer guaranteed." Important nuance If you're seeing this error even when you'd expect capacity to be available, there are a few possibilities: - you're on a Prefect version where this behavior was older/incomplete - another run took the slot after lease expiry - worker/orchestration timing caused the transition check to fail - the concurrency limit is very tight relative to infra scheduling delays Practical takeaway Prefect does try to do what you're suggesting, but only at the

PENDING -> RUNNING

transition, and only if it can reacquire the slot without violating the limit. If you want, I can help verify whether your specific Prefect version includes this reacquisition behavior and point to the exact code path / release where it was added.

Yes — in 3.7.7, Prefect does try to reacquire the slot before allowing

PENDING -> RUNNING

. So if you're still seeing an error, the important clarification is: Prefect is not failing because it forgot to try. It's failing because the reacquisition attempt did not succeed. What 3.7.7 actually does At

PENDING -> RUNNING

, Prefect: 1. looks at the lease ID from the pending state 2. tries to renew that lease 3. if the lease is gone, it tries to acquire a fresh slot 4. if that fresh acquisition succeeds, the run starts 5. if it fails, the transition is rejected and the run is set to

Cancelled

The failure case in 3.7.7 is effectively:

text
Deployment concurrency slot lost during provisioning - no slots available to continue execution

with the rejection reason:

text
Deployment concurrency limit reached after lease expiry.

So why would reacquisition fail? Because by the time the run is finally runnable, Prefect checks the concurrency limit again. If all slots are occupied at that moment, it must reject the run. Example with limit = 5: - 5 runs get leases and go

PENDING

- one of them is stuck because no node has enough memory - its lease expires - the slot becomes available again - another run grabs that slot - later, the original stuck run finally gets a chance to start - Prefect tries to reacquire - but all 5 slots are now taken - result: cancellation/error So the reacquisition logic is best effort, not a reservation guarantee. Why this still feels surprising Because from your perspective, the run was "already admitted once", so you'd expect it to keep priority. But once the lease expires, Prefect treats that slot as no longer reserved. After that point, the run has to compete again. Answer to your original "why can't it just reacquire?" It can and does in 3.7.7. If you still get the error, that means one of these is true: - the slot was no longer available when start was attempted - the worker took too long to reach

RUNNING

- another run claimed the freed capacity first What this means operationally If the real issue is "worker/infrastructure can't place the run because of memory pressure", then deployment concurrency is not enough to protect you from this outcome. The run is losing its reservation while waiting for infra capacity, and reacquisition is only successful if capacity is still free later. So if you want to avoid this without increasing grace period or adding resources, the real levers are: - reduce how long runs stay pending - reduce per-run memory needs - change concurrency strategy so these runs don't sit pending holding deployment slots - separate infra-capacity limits from business concurrency limits If you want, I can help you reason through a concrete example of your setup (for example Kubernetes worker + work pool + deployment concurrency=5) and explain exactly how a run can end up cancelled even on 3.7.7.