Yes — the 5 minute behavior is expected from Prefect’s deployment concurrency lease system. Most likely, your issue is caused by a combination of: * version skew between server and worker * server:

3.7.8

* worker:

3.4.1

* possibly HA lease-storage/configuration concerns The biggest red flag is that your worker is older than the minimum version needed for reliable deployment concurrency lease handling. What’s happening Prefect deployment concurrency works by creating a lease for the running flow. That lease has a default duration of 300 seconds = 5 minutes. The worker is supposed to renew that lease before it expires. If renewal fails: * the server assumes the slot is no longer reserved * the concurrency slot is released * other queued runs may proceed or state transitions may behave unexpectedly * you see:

Concurrency lease renewal failed - slots are no longer reserved. Execution will continue, but concurrency limits may be exceeded.

So yes: there is a configuration/default that explains the 5-minute timing. Relevant default The server-side setting is:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION

Default:

300

seconds That aligns exactly with what you’re observing. Why your setup is especially susceptible Your worker is on

3.4.1

, while your server is

3.7.8

. Based on the Prefect 3.x code, deployment concurrency lease support expects newer clients/workers for proper lease maintenance. In particular,

3.4.11+

is the important threshold for more reliable lease renewal behavior. So with `3.4.1`: * the worker may not renew leases correctly/reliably * after ~5 minutes the server expires the lease * subsequent runs may get cancelled or concurrency may be misapplied Recommended fix 1. Upgrade your workers first Best fix: run workers on the same version as the server, ideally

3.7.8

. At minimum, upgrade beyond

3.4.11

, but matching server/worker versions is strongly recommended. 2. Check HA lease storage In HA, concurrency lease state must be shared correctly across all API/server instances. If you are running multiple Prefect server replicas, make sure lease storage is not effectively local/in-memory per replica. If needed, review/configure:

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

For HA, use shared storage such as filesystem-on-shared-volume or Redis-backed lease storage, depending on your deployment model. 3. As a temporary mitigation, increase the lease duration You can increase the initial lease TTL to reduce premature expiration risk:

bash
export PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=600

That changes the lease to 10 minutes. This is only a mitigation — not the real fix if the worker cannot renew leases properly. Important note about cancellations With deployment concurrency limit

1

, subsequent runs should normally wait rather than behave strangely. If they are being cancelled, that suggests the lease/orchestration state is getting inconsistent, which again points back to the old worker version and/or HA coordination issues. What I’d do next 1. Upgrade worker(s) to

3.7.8

2. Confirm all server replicas share the same lease storage/backend 3. If needed, temporarily raise:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION

4. Re-test by launching multiple runs simultaneously Useful references * Prefect deploy docs: Deploy via Python * Prefect settings docs: Settings reference * Prefect repository: Prefect GitHub If you want, I can help you narrow this down further by checking: * how your HA server is deployed * what lease storage backend you’re using * whether the runs are truly Cancelled vs Crashed vs Late vs being replaced by orchestration decisions