Thanks for the detail — I get how frustrating that is. What you’re seeing is a known edge case with isolated containers: when the process is OOMKilled, it can’t report a final state, so task runs can remain “RUNNING” and hold tag-based concurrency slots until the server reclaims them. What happens under the hood (Prefect 3.14) - Tag-based concurrency uses leases. If the flow/task process dies, the lease renewal stops and the server’s cleanup service will revoke those leases automatically after their TTL (typically ~5 minutes). When that revocation happens, the slots are freed even if the UI still shows the task as RUNNING for a while. - In a per-flow Docker container pattern, there’s no external runner to mark the run “CRASHED” when the process disappears; cleanup is driven by lease expiration. - If your slots are not freeing within a few minutes after the crash, that’s abnormal and worth digging into (see checks below). Immediate ways to unblock - Free the tag’s slots now:

prefect concurrency-limit inspect <your-tag>
prefect concurrency-limit reset <your-tag>

This clears the active_slots for that tag immediately so your work can continue. - Manually fail the stuck flow run(s) to trigger all of the normal cleanup:

prefect api POST /flow_runs/<flow-run-id>/set_state --data '{"state": {"type": "FAILED", "message": "Manually failed after container OOMKilled"}}'

Recommended mitigations going forward - Expect a short delay for automatic cleanup: After an abrupt crash (like OOM), concurrency slots should free automatically once the lease expires (commonly ~5 minutes). If you’re on Prefect Cloud, this timing is managed for you. If you’re self-hosting, confirm the repossessor service is enabled and running. - Add timeouts: - Task timeouts ensure long-running tasks can’t hold slots forever if they hang (this won’t catch OOM, but it prevents other “stuck” states). - Flow-level

timeout_seconds

also puts an upper bound on a run. - Consider a cleanup automation (Cloud): Create an automation that detects a flow run that entered RUNNING but has not changed state within X minutes and then sets it to FAILED. This ensures slots free even if infra kills the container. - Right-size container memory/limits and add retries/backoff to reduce OOMs and make recovery predictable. Troubleshooting checklist (if slots don’t free automatically within ~5–6 minutes) - Do you see the issue with all tags or only specific ones? - How long do the slots remain occupied after the crash? - Are you on Prefect Cloud or self-hosted Server? If self-hosted, verify the background “repossessor” service is enabled and logs show expired lease revocations around the time of the crash. - After an OOM, does

prefect concurrency-limit inspect <tag>

still show active_slots for many minutes? If yes, resetting the limit is a safe immediate fix; share one of those task run IDs and we can help trace the lease cleanup path. Quick commands you can use - Inspect/reset tag limits

prefect concurrency-limit ls
prefect concurrency-limit inspect <tag>
prefect concurrency-limit reset <tag>

- Mark a stuck run as failed (to release all its slots)

prefect api POST /flow_runs/<flow-run-id>/set_state --data '{"state": {"type": "FAILED", "message": "Manually failed after container OOMKilled"}}'

A couple of clarifying questions so I can tailor this: - Are you running on Prefect Cloud or self-hosting Prefect Server? - Roughly how long did the tag stay “occupied” after the OOM event — minutes or hours? - If you run

prefect concurrency-limit inspect <your-tag>

right after an OOM, do you see task run IDs in

active_slots

remain for >5 minutes? If you can share one example flow run ID and the tag name, I can outline exact steps to clean up and confirm whether automatic lease revocation is working as expected.

Got it — thanks for confirming you’re self‑hosted. If a tag has been occupied for 18h, automatic lease cleanup is not running as expected. Let’s unblock you now, then fix the root cause. Immediate unblocking - Free the tag’s slots right away:

prefect concurrency-limit inspect <your-tag>
prefect concurrency-limit reset <your-tag>

- Mark the crashed flow run as failed so orchestration cleanup runs:

prefect api POST /flow_runs/<flow-run-id>/set_state --data '{"state": {"type": "FAILED", "message": "Manually failed after container OOMKilled"}}'

- If you want to clean up individual task runs too (optional):

prefect api POST /task_runs/<task-run-id>/set_state --data '{"state": {"type": "FAILED", "message": "Infra crash"}}'

What likely caused the 18h “stuck” slot - In self-hosted Server, the repossessor service is responsible for revoking expired concurrency leases. If it’s disabled/not running, tag slots will never be freed after an abrupt crash (like OOM). - By default, leases should expire and be cleaned within ~5 minutes. If they didn’t, repossessor is probably off or failing. Verify and fix on your server - Ensure the repossessor and foreman services are enabled (defaults are true). Set these env vars on the Prefect Server process (API container/pod):

PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED=true
PREFECT_SERVER_SERVICES_REPOSSESSOR_LOOP_SECONDS=15
PREFECT_SERVER_SERVICES_FOREMAN_ENABLED=true
PREFECT_SERVER_SERVICES_FOREMAN_LOOP_SECONDS=15
PREFECT_SERVER_SERVICES_FOREMAN_INACTIVITY_HEARTBEAT_MULTIPLE=3
PREFECT_SERVER_SERVICES_FOREMAN_FALLBACK_HEARTBEAT_INTERVAL_SECONDS=30

- Restart the server and check logs for messages like “Starting repossessor service”. If needed, set

PREFECT_LOGGING_LEVEL=DEBUG

to confirm it’s looping. - Optional: if you want faster automatic cleanup after crashes, you can shorten the default lease duration:

PREFECT_SERVER_CONCURRENCY_INITIAL_DEPLOYMENT_LEASE_DURATION=300  # default 300s; can reduce, min 30s

Reduce risk going forward - Add timeouts to tasks/flows so genuinely stuck work doesn’t hold slots forever:

@task(timeout_seconds=900)
def my_task(...): ...

- If you want proactive cleanup on self-hosted, run a small cron/Job that finds stale RUNNING flow runs and marks them FAILED via the API (happy to share a short script). - If you use Docker work pools for isolated containers, keep in mind OOMKills can’t report final state; the repossessor is the safety net that must be running. To help pinpoint exactly what’s wrong, could you share: - Prefect Server version (from the server container:

prefect version

) - Whether your server logs show the repossessor service starting - Output of:

prefect concurrency-limit inspect <your-stuck-tag>

(including the

active_slots

list and one task run ID) Once we see that, we can confirm whether repossessor/lease cleanup is the culprit and get you a precise fix.