Happy to help troubleshoot this. Assuming you’re on Prefect 3.x: concurrency slots are managed via a lease under the hood and should be released when the

concurrency

context exits (or eventually when the lease expires). On self‑hosted servers, the usual culprits are: processes exiting before cleanup, network hiccups to the server during release/renewal, or using the legacy tag-based limits in ways that don’t match how your code is acquiring slots. A few quick checks to pinpoint what’s happening: - Are your limits tag-based (set via CLI) or name-based (used in code)? - Tag-based limits use the CLI group

prefect concurrency-limit

. - Name-based limits are used in code via

prefect.concurrency.sync/asyncio.concurrency("my-limit")

. - Do slots eventually release after a fixed time (e.g., ~5 minutes), or do they remain stuck indefinitely? - Releasing after a fixed time suggests the worker/process died or could not reach the server to release, so the lease expired. - Stuck indefinitely suggests misconfiguration or a bug. - What lease storage backend is your self-hosted server using (memory/filesystem/Redis)? Redis is recommended for reliability at scale. - Which Prefect version is running on your server and workers? Please share the output of

prefect version -v

from both. Immediate actions you can take 1) Inspect the current concurrency state - List limits and check who’s holding slots:

prefect concurrency-limit ls
prefect concurrency-limit inspect <tag>

inspect

will show

active_slots

and the TaskRun IDs currently holding them. 2) If you used the concurrency context in code, confirm it always exits - Sync example:

from prefect.concurrency.sync import concurrency

def run_work():
    with concurrency("my-limit", occupy=1, strict=False):
        # your work
        ...

- Async example:

from prefect.concurrency.asyncio import concurrency

async def run_work():
    async with concurrency("my-limit", occupy=1, strict=False):
        # your work
        ...

Tip: If there are early returns, exceptions, or cancellations, make sure they pass through the context manager (or wrap your work in try/finally) so release is attempted. 3) Check worker/server logs around the time slots “stick” Look for messages about: - Unable to acquire/release a concurrency slot - Lease not found or lease renewal failures - Network timeouts to the API 4) If you need to unblock immediately (tag-based limits) - You can reset a tag-based limit to clear stuck active slots. Use with care; this overrides current holders:

prefect concurrency-limit reset <tag>

Common root causes on self‑hosted - The worker process was killed or crashed while holding slots; release didn’t complete. Leases should eventually expire; if they don’t, check server/clock skew and upgrade to the latest 3.x. - Using in-memory or filesystem lease storage on a multi-instance server can cause inconsistencies under load. Prefer Redis for lease storage in production. - Code paths that never exit the concurrency context (e.g., background threads or tasks that bypass the `with`/`async with` scope). - Older versions: there have been reliability fixes to concurrency/lease release in recent 3.x releases—upgrading often resolves sticky slots. Could you share: - Your Prefect server and worker versions (

prefect version -v

) - Whether you’re using tag-based limits or

prefect.concurrency.*

in code (plus a snippet showing how you acquire the slot) - Whether slots eventually release after a fixed duration - Your server’s lease storage backend With those details, I can pinpoint the cause and recommend a precise fix.

Thanks for the details — that helps narrow it down. A few key points based on what you shared: - You’re using tag-based limits (managed via the

prefect concurrency-limit

CLI). - Slots don’t auto-release; when you manually clear them they work once or twice, then get stuck again. - You can’t run

prefect version

. Direct answers and next steps 1) “What is my server lease?” - Each active slot is represented by a lease stored by the server. The “lease storage backend” determines how/where these are kept (in-memory, filesystem, or Redis). - On a self-hosted server, you can check which backend you’re using by inspecting the server’s environment or settings for: -

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

- If you’re running via Docker/Compose/K8s: - Inspect the server container/pod env: look for

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

- If it’s unset, the server is likely using an in-memory/default backend. For production, using Redis is recommended because it avoids various race/consistency issues that can make slots appear “stuck.” If you share how you launch Prefect Server (docker-compose, Helm, systemd, etc.), I can give the exact command to check. 2) Quickly inspect who is holding your tag-based slots - Identify the tag and run:

prefect concurrency-limit inspect <your-tag>

- This shows

active_slots

and the TaskRun IDs currently holding them. After you run a few tasks, run the command again to see if: - The same TaskRun IDs persist after the runs complete (suggests release isn’t firing on state change) - New TaskRun IDs accumulate (suggests repeated acquisition without corresponding release) 3) If you need to unblock immediately - You can reset the tag-based limit (use with care; it force-clears all slot holders):

prefect concurrency-limit reset <your-tag>

4) Why this pattern happens (likely causes for tag-based limits on self-hosted) - The server can’t reconcile/release leases reliably if the backend is volatile or under concurrency stress (e.g., in-memory or filesystem backends under load). Redis is much more reliable for production. - Workers or tasks crash or get cancelled before the server runs the release step associated with leaving the RUNNING state. - The same task is retried or mapped multiple times and multiple leases accumulate; if the server doesn’t find and revoke each matching lease for that TaskRun, you can see exactly what you described — it “releases” once, then gets stuck on subsequent uses. 5) Since

prefect version

isn’t working - Try:

prefect --version

- Or from Python:

python -c "import prefect; print(prefect.__version__)"

- If those fail too, let me know the error — it may point to an environment or PATH issue. To help me get you to a precise fix, could you share: - The tag name and outputs of: -

prefect concurrency-limit inspect <tag>

before/after a run where it gets stuck - How your server is deployed (docker-compose, Helm, bare process) and whether you can see

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

in the server environment - How the tag is applied (task decorator tags, inherited from flow, etc.) - Whether the task(s) with that tag are retried or mapped If you’d like, I can also suggest an exact Redis-backed configuration once I know how you’re hosting the server.

Great question — and good instinct to ask about Redis. - Prefect does not store concurrency leases in the database. The DB only stores the limit definitions and counters; the actual lease records live in a separate “lease storage” backend. - If you haven’t configured lease storage, the server defaults to an in-memory backend. That works for a single API process, but in multi-instance or load-balanced setups it can cause exactly what you’re seeing: a slot is acquired on one API instance and the release later hits a different instance that can’t find the original lease record. The result looks like “it releases once or twice, then gets stuck.” Short answer: yes — for a self-hosted, production/HA setup, using Redis for lease storage is strongly recommended. How to switch your server to Redis lease storage 1) Install the integration on the server image/process

pip install "prefect-redis"

2) Point the server at the Redis-backed lease storage - Set this on the API server container/pod:

export PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE="prefect_redis.lease_storage"

3) Configure Redis connection - If you already use Redis for Prefect messaging, the same connection is typically reused. - Otherwise, set your Redis connection env vars (e.g., URL/host/port) according to your environment and how you install prefect-redis. 4) Restart the Prefect server and verify

prefect config view

- Look for

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE = prefect_redis.lease_storage

5) Clear any currently stuck tag-based leases - After switching, reset affected tags once to start clean:

prefect concurrency-limit reset <your-tag>

Why Redis helps - Centralized, atomic lease operations across all API instances - Avoids the per-process visibility problems of in-memory storage - More robust under concurrency and restarts A couple of quick confirmations that will help me tailor exact steps: - Are you running multiple API server instances (e.g., behind a load balancer or in Kubernetes)? - How are you deploying the server (docker-compose, Helm, bare process)? - Do you already have Redis running for other Prefect features? Once I know your deployment style, I can provide exact env var snippets and where to place them.