Short answer: that message means a concurrency lease that your run was holding got lost/expired during renewal. Even if you didn’t set a limit on the flow, a task inside the flow likely picked up a tag-based concurrency limit automatically, or your code used a global concurrency limit, and the lease for that limit could not be renewed. Why this can happen - Tag-based limits apply automatically to tasks: If any task has tags (e.g.,

@task(tags=["db","prod"])

) and your workspace has a concurrency limit for any of those tags, Prefect will acquire a lease for each matching tag while that task runs. If the lease can’t be renewed (expired, revoked, limit deleted/reset, network hiccup), you’ll see this error and the task/run will be terminated to avoid over-allocation. - A global limit was used in code: If you (or a library you use) wrapped code with

prefect.concurrency.concurrency(...)

, that also acquires/renews a lease. Renewal failures will terminate execution. - The limit was changed mid-run: Deleting/resetting a limit while there are active leases will cause renewals to fail for those runs. - Transient issues: Loss of connectivity to Prefect (Cloud or server) or worker/container restarts can cause renewal to fail. How to confirm quickly - List existing tag concurrency limits:

prefect concurrency-limit ls

- Inspect a specific limit to see active slots (which task runs are holding leases):

prefect concurrency-limit inspect <limit_name>

- Check your code for: - Task or flow tags (

@task(tags=[...])

,

with tags(...):

) - Any usage of

from prefect.concurrency import concurrency

(or

.sync/.async

variants) - Turn on debug logs for more detail: set

PREFECT_LOGGING_LEVEL=DEBUG

and re-run to see which limit name was involved. Common fixes - If a tag limit is unintentionally catching your tasks, remove the limit or rename/remove the tag on those tasks. - Avoid deleting/resetting limits while runs are active; disable them instead, then drain. - For manual/global limits, increase

lease_duration

to exceed your longest critical section and avoid tight renew windows. - If you don’t want execution to stop on renewal failure for manual limits, ensure

strict=False

when using

concurrency(...)

. A couple of clarifying questions to help pinpoint the cause: - Do any of your tasks or flows have tags like “prod”, “db”, “api”, etc.? - Does

prefect concurrency-limit ls

show any active limits that might match those tags? - Did anyone recently delete or reset a concurrency limit while runs were in progress? - Are you using

prefect.concurrency.concurrency(...)

anywhere in your code? If you can share a flow run ID or a snippet of the run logs around the error, I can help identify the exact limit/name that triggered the termination.

Got it — thanks for checking tags and code usage. This specific message is most often from deployment-level concurrency, not task/tag concurrency. In Prefect 3.x, when a deployment has a concurrency setting, the flow engine maintains a deployment concurrency lease in the background. If that lease can’t be renewed (expired, revoked, or lost on the server), the engine logs exactly that message and cancels the run to avoid over-allocation. Important distinctions: -

prefect concurrency-limit ls

only shows task/tag-based limits. It does not show deployment-level concurrency. - No task tags and no

concurrency(...)

usage rules out task/global limits, but a deployment with concurrency enabled will still create and renew a lease for each flow run. How to verify - Check the deployment for a concurrency setting (UI is the clearest: Deployment > Settings > Concurrency). - Or via CLI:

prefect deployment inspect "<flow_name>/<deployment_name>"

Look for a concurrency-related field (e.g., a concurrency section/options) in the JSON. Common reasons leases fail even when you didn’t intend to use limits - Deployment concurrency is/was configured (possibly set earlier) and a run still holds a lease. - The server restarted while using in-memory lease storage (OSS default), which drops leases; renewals then fail. - The concurrency setting was changed/removed while runs were active. - Transient network/API issues caused the renewal retries to exhaust. Next steps - If you don’t want deployment-level concurrency, remove or disable it on the deployment and redeploy. - If you do want it: - Avoid deleting/changing it while runs are active. - If you’re on OSS server, consider persistent lease storage to survive restarts:

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect.server.concurrency.lease_storage.filesystem

- If you suspect transient issues, re-run with debug logs to see which lease is involved:

PREFECT_LOGGING_LEVEL=DEBUG

If you share a flow run ID and the deployment name, I can confirm whether a deployment concurrency lease was attached and why the renewal failed. Are you running in Prefect Cloud or a self-hosted server, and did the server/agent restart around the time of these cancellations?

Thanks — that snippet shows a deployment-level concurrency limit that Prefect creates automatically for the deployment with ID 656afbd5-05bb-4623-a03c-e8247a68f114. Its internal name is deployment:656afbd5-05bb-4623-a03c-e8247a68f114 and it’s why the engine is maintaining a lease and killing runs when renewal fails. You have a few ways to remove/disable it: Recommended: remove concurrency from the deployment (prevents it from coming back) - In the UI: Deployment > Settings > Concurrency, clear the limit (set to None) and save. - Or redeploy with concurrency_limit=None:

from prefect import flow

  @flow
  def my_flow():
      ...

  my_flow.deploy(
      name="your-deployment",
      work_pool_name="your-pool",
      concurrency_limit=None,  # remove deployment concurrency
  )

- If you use prefect.yaml, ensure the deployment entry does not set concurrency_limit (or set it to null), then run:

prefect deploy

Quick CLI actions against the existing limit - Disable it now (stops enforcement without deleting the record; safer for active runs):

prefect global-concurrency-limit disable "deployment:656afbd5-05bb-4623-a03c-e8247a68f114"

- Delete it (removes the record immediately):

prefect global-concurrency-limit delete "deployment:656afbd5-05bb-4623-a03c-e8247a68f114"

Note: Deleting while runs are active will cause their lease renewals to fail (the exact error you saw) and those runs may be canceled. If you need to avoid that, disable first, let runs finish, then redeploy with concurrency_limit=None so it doesn’t get recreated. Verify - List all global limits:

prefect global-concurrency-limit ls

- Inspect that specific limit:

prefect global-concurrency-limit inspect "deployment:656afbd5-05bb-4623-a03c-e8247a68f114"

- After removal, confirm it no longer appears and that your deployment no longer shows a concurrency setting (UI or

prefect deployment inspect "<flow>/<deployment>"

). If you want, share the deployment name and I can suggest the exact redeploy command/snippet to remove it cleanly.

Thanks for the update. Two important clarifications that explain what you saw: - Omitting concurrency on redeploy does not remove it. In Prefect 3.x, if you redeploy and do not pass a concurrency_limit, the existing deployment concurrency is left unchanged. That’s why it was still there after your redeploy. - To remove it via redeploy, you must explicitly set concurrency_limit=None. That translates to a “clear this field” on the server and deletes the linked global limit named deployment:<deployment_id>. Practical options going forward - Remove it during redeploy (recommended so it doesn’t get recreated):

my_flow.deploy(
      name="your-deployment",
      work_pool_name="your-pool",
      concurrency_limit=None,  # explicitly clear
  )

Or in prefect.yaml set:

concurrency_limit: null

then run:

prefect deploy

- If you need to act immediately on an existing limit: - Disable enforcement:

prefect global-concurrency-limit disable "deployment:656afbd5-05bb-4623-a03c-e8247a68f114"

- Or delete it (as you did):

prefect global-concurrency-limit delete "deployment:656afbd5-05bb-4623-a03c-e8247a68f114"

Tip: After you clear it via redeploy, verify it’s gone and won’t be recreated: - Check the deployment detail (UI or)

prefect deployment inspect "<flow>/<deployment>"

- List global limits:

prefect global-concurrency-limit ls

If you still see it return after a redeploy with concurrency_limit=None, let me know the deployment name and how you’re deploying (Flow.deploy, from_source().deploy, or prefect.yaml), and I’ll help trace the exact payload being sent.

Thanks for the extra detail — that helps narrow it down. What likely happened - That run was created/scheduled while deployment-level concurrency was still enabled. When a run is created with deployment concurrency, it gets a deployment_concurrency_lease_id attached in its state details. - After you removed the limit and especially after you deleted the global limit record, that in-flight run still tried to renew its existing lease. Because the backing limit/lease no longer existed, the renewal returned 410 and the flow engine terminated with: Concurrency lease renewal failed - slots are no longer reserved. Terminating execution to prevent over-allocation. This exact message only fires in the flow engine’s deployment-lease maintainer (strict mode), not in task/tag concurrency. So even though you cleared concurrency on the deployment, any already-created run that carried a lease will still try to renew it and will crash if the limit was deleted. Why no extra logs with PREFECT_LOGGING_LEVEL=DEBUG - That message is logged at error level already; DEBUG won’t add much around this specific failure. - In ECS, make sure the env is applied on the flow’s task container (deployment’s job variables env is correct), and in the UI filter “All log levels” to see debug logs. But for this path, the single error line is expected. How to get unstuck right now - Cancel any runs that were created before concurrency was removed or while you were deleting the limit, then trigger fresh runs. New runs created after redeploy with concurrency_limit=None will not acquire a deployment lease and should not hit this error. - If you still see the error on brand-new runs, it means they are not using the updated deployment (e.g., a different deployment is being triggered). Open the failing run in the UI and confirm: - The “Deployment” link matches the deployment you redeployed. - In the run’s JSON, state_details does not include deployment_concurrency_lease_id. Safer removal pattern for next time - Instead of deleting the limit while runs are in-flight, first disable it:

prefect global-concurrency-limit disable "deployment:<deployment_id>"

Then redeploy with concurrency_limit=None so new runs won’t acquire a lease. Let the old runs finish (their lease will still renew because the lease object still exists), then you can delete the limit if you want. - Important: Omitting concurrency_limit on redeploy leaves the old setting in place; you must explicitly set concurrency_limit=None to remove it. Useful checks you can run in your workspace - Verify the run’s details (you’ll need to run this in your own environment):

prefect flow-run inspect 069b0361-3271-73ae-8000-3591a504f409

Look for state_details.deployment_concurrency_lease_id. If present, that confirms this was a carry-over lease. - List global limits to ensure nothing reappeared:

prefect global-concurrency-limit ls

Can you confirm: - Whether this failing run was created before you redeployed with concurrency_limit=None? - Whether new runs (created after the redeploy) still show this error? - In the failing run’s JSON, does state_details include deployment_concurrency_lease_id? If a brand-new run (post-redeploy) still fails, share the run’s deployment name and I’ll help double-check the deployment config and the exact cleanup steps to ensure no hidden limit is being recreated.

Thanks for the logs. “Heartbeat loop cancelled / Stopped flow run heartbeat context” is a side-effect of the engine shutting down; it doesn’t tell us why. The root cause is still the deployment-level concurrency lease maintainer cancelling the run. Given that this run was after you deleted the global concurrency limit, the only way the engine emits that concurrency message is if the flow run still had a deployment_concurrency_lease_id in its state details. That happens if: - The run was created when deployment concurrency was still enabled (it acquired a lease at scheduling/start), or - The deployment still had concurrency enabled when this run was created (e.g., you deleted the limit record but the deployment config still has a concurrency limit, so orchestration re-acquired a new lease), or - You triggered a different deployment (same flow/name) that still has concurrency. What I’d like to check to pinpoint it 1) Inspect the failing run’s state details to confirm it had a lease: - In your workspace:

prefect flow-run inspect 069b0361-3271-73ae-8000-3591a504f409

Look for state_details.deployment_concurrency_lease_id. If it’s present, that’s why the maintainer ran and cancelled the run when renewal failed. 2) Verify the deployment that created this run truly has no concurrency: - In UI: Deployment > Settings > Concurrency should show “Not limited” - Or:

prefect deployment inspect "<flow_name>/<deployment_name>"

Confirm it does not show a concurrency section and that global_concurrency_limit is null. 3) Double-check there isn’t another deployment being used: - In the run details (UI), click the Deployment link and verify its ID matches the one you updated. - It’s easy to have two deployments with the same name in different projects/workspaces or created by different pipelines. 4) Ensure no global limit got recreated:

prefect global-concurrency-limit ls

Confirm there is no entry named deployment:<deployment_id>. Why this can still happen “after deletion” - If the run was created (or moved to Running) while the deployment still had a concurrency limit, it already holds a lease. Deleting the limit afterward makes the next renewal return 410; the engine cancels with the exact message you saw. - If the deployment still carried a concurrency limit (because it was omitted on redeploy), the orchestrator will acquire a new lease for new runs, even if you previously deleted the limit record. Safe cleanup pattern - If you need existing in-flight runs to finish, don’t delete the limit record immediately. First disable it:

prefect global-concurrency-limit disable "deployment:<deployment_id>"

Then redeploy with concurrency_limit=None so new runs won’t acquire a lease. Let old runs finish (their lease can still renew). Finally, delete the limit if you want it gone. - If you already deleted it and a run is still failing, cancel that run and trigger a fresh one created after the redeploy that explicitly set concurrency_limit=None. ECS logging note - The concurrency-maintainer logs at error on failure, so setting PREFECT_LOGGING_LEVEL=DEBUG won’t add many extra lines around this path—what you saw (heartbeat loop cancelled) is typical when the engine shuts down. - Make sure the env var is applied to the ECS task’s container (deployment job variables is the right place). If you can paste the output of: - prefect flow-run inspect 069b0361-3271-73ae-8000-3591a504f409 (state_details section) - prefect deployment inspect "<flow>/<deployment>" (concurrency fields) I’ll call out exactly where the lease is coming from and the next exact step to clear it.