Short answer: not necessarily at submission time. With deployment concurrency

limit=1

and

CANCEL_NEW

, Prefect tries to prevent the new run from proceeding, but the error you saw means the run had a concurrency lease at some point, then lost it later while execution was starting or running:

Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation.

So this message is basically Prefect saying:

“This run no longer owns the concurrency slot, so I’m stopping it now to avoid violating the limit.”

What

CANCEL_NEW

means - If a new run cannot get a slot, it should be cancelled instead of waiting. - But if a run already got a lease and then that lease is lost/revoked/expired, the behavior is different: the worker/engine aborts execution with the lease-renewal error you saw. So yes, it is being cancelled/directly terminated, just at the moment Prefect realizes the lease is gone, not always earlier. Why this can happen Common reasons: - the lease expired before it was renewed - the server revoked/released the lease - the run spent too long between provisioning and actual execution - temporary server/API issues caused renewal to fail - another concurrency transition caused the slot to no longer belong to that run Important distinction There are two phases: 1. Acquiring the deployment concurrency slot 2. Maintaining/renewing the lease while the run proceeds

CANCEL_NEW

mainly affects phase 1. Your error is from phase 2. What I’d check - Are multiple runs being triggered almost simultaneously? - Is there a long delay between a run being submitted and actually starting on the worker? - Any worker / API connectivity interruptions? - Any server logs showing lease expiration or revocation? If you want, I can help you narrow this down by looking at: - how the deployment is triggered - worker type - whether this is Cloud or self-hosted - approximate timing between the first run start and the second run submission Relevant docs: - Deployments - Deployment concurrency limits

Yes — that timing explains it. Your run spent about 9 minutes between: -

submitting flow run

at

09:29:55

- actual flow start at

09:38:47

That is very likely longer than the deployment concurrency lease was able to survive. What happened The deployment slot was probably acquired when the worker started submission, but the process/opening/provisioning took too long: -

Opening process...

started at

09:38:26

- flow began at

09:38:47

- lease renewal failed immediately at

09:38:48

Prefect uses a renewable lease for deployment concurrency. If the slot is no longer reserved by the time the run is actually executing, Prefect kills the run to avoid exceeding

concurrency_limit=1

. So in your case: 1. run got the slot 2. startup/provisioning delayed too long 3. lease expired or was revoked before/around actual execution 4. when Prefect tried to renew/validate it, it was gone 5. run was terminated with

Crashed('Execution was cancelled by the runtime environment.')

Why

CANCEL_NEW

didn’t prevent this earlier Because

CANCEL_NEW

only governs what happens when a new run tries to acquire a slot and none are available. Your run appears to have already acquired a slot earlier in the lifecycle. The failure came later, during lease maintenance/validation, so the result is a crash/termination rather than a clean “cancel before start”. The strongest signal in your logs This gap:

09:29:55 submitting flow run
09:38:26 Opening process...
09:38:47 Beginning flow run

That ~9 minute delay is the smoking gun. What to investigate next For a

ProcessWorker

, I’d look at why process startup is taking so long: - machine/resource pressure on the worker host - many queued local processes - blocking deployment steps - slow filesystem / network mounts - expensive

pull

or setup steps before the flow starts - worker host being CPU or memory constrained What you can do 1. Reduce startup delay - simplify deployment steps - avoid slow setup before flow code starts - make sure the worker machine is healthy 2. Check whether multiple runs are contending - with

limit=1

, any overlap makes timing more sensitive 3. Consider whether deployment concurrency is the right control point - if the true concern is preventing overlap of a specific resource, sometimes a global concurrency limit is a better fit than deployment concurrency - docs: Global concurrency limits 4. Review worker and API/server health - if this is self-hosted, check API/server logs for lease expiration/revocation - if Cloud, inspect worker host performance and connectivity One subtle point This may look like a bug because the run starts and then dies immediately. But based on these timestamps, it’s consistent with: - lease acquired too early - execution starting too late - lease no longer valid at run start If you want, I can help you determine where that 9-minute delay is coming from for

ProcessWorker

and suggest the most likely causes in order.