Good morning! That message comes from Prefect’s global concurrency system. In short: your flow acquired one or more concurrency “slots”, but later the process stopped renewing the lease on those slots. When a lease can’t be renewed, Prefect assumes the slots are no longer reserved and cancels the run to prevent over-allocation — even if your tasks already completed. That’s why you can see “all tasks succeeded” yet the flow run is marked as failed/crashed. What this means technically - Prefect uses leases to reserve concurrency slots while a flow/task is running. A background loop keeps renewing those leases with the API. - If the renewals don’t reach the API in time (e.g., transient network issue, process pause, container throttling, worker restart, long GC pause, DNS hiccup), the lease expires and Prefect cancels the execution with: Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. - This typically only appears when you’ve enabled global concurrency limits/rate limits for your flow/task tags. References - Global concurrency limits: Docs: Global concurrency limits - Settings reference (logging, etc.): Docs: Settings reference - API for lease renewal: API: renew_concurrency_lease - Related GitHub issues (same message/symptoms): - #18893 Concurrency lease renewal failures causing flow crashes - #18627 Future.CancelledError at lease renewal callback after flow finished Common causes we’ve seen - Brief network blips between the runner and the API (Cloud or OSS Server) - Worker pod/container restart or CPU throttling that pauses the Python process - Long GC pauses or blocking calls stalling the background renewal loop - DNS/resolver hiccups - Clock skew between the run environment and the API (less common but can shorten effective TTL) - Recent upgrades where the worker image and environment changed timings/behavior How to confirm it’s this - Check whether the affected flow or any of its tasks use global concurrency limits (via tags/rules). If there’s no concurrency rule attached, this message shouldn’t appear. - In the flow-run logs, you’ll see the message right near the end of execution. If you’re on Kubernetes, also check pod logs to confirm the container kept running while Prefect marked the run failed. - Enable debug logs to get more detail:

PREFECT_LOGGING_LEVEL=DEBUG

- If you self-host, also look at API server logs around the same timestamps for lease renewals. Mitigations and next steps - If you can, temporarily disable the relevant concurrency rule(s) on this deployment to verify the root cause. If the failures disappear, it’s almost certainly lease renewal timing. - Ensure the runner/worker environment is healthy: - No CPU throttling or OOM kills - Stable network/DNS to the Prefect API - System time synced (e.g., NTP) - Upgrade your worker/runtime to the latest Prefect 3.x patch (we continually harden concurrency/runner behavior). Release notes: Prefect releases - Increase visibility: - Run with DEBUG logging as above. - If you’re on Kubernetes, check for restarts/retries and resource throttling. - If you must keep concurrency limits but this is happening intermittently, you can: - Reduce environmental causes (network/CPU) as above. - Consider relaxing how broadly/strictly rules apply (e.g., narrower tags) so fewer contracts are active. - As a short-term workaround for validation, run a few times without the rule to confirm the error disappears. A few questions to help us pinpoint it: - Are you on Prefect Cloud or self-hosted Server?

- Prefect version of your worker/runtime image? - What worker type are you using (Kubernetes, Docker, Process, etc.)? - Which global concurrency rules are applied (tags and limits)? - Roughly when in the run does the message appear (right at the end vs. mid-run)? - Any recent changes (upgrades, scaling, resource limits) around when this started? If you share a snippet of the logs around the error and your worker/deployment config (without secrets), I can suggest more targeted steps.