<@ULVA73B9P> I believe concurrency leases get stuc...
# ask-marvin
m
@Marvin I believe concurrency leases get stuck and essentially concurrency limits reduce over time. Is there a way to confirm this and or reset or other way to fix this issue
m
thought for 301.8 seconds
Short answer: yes—you can confirm whether leases are stuck and you can reset them. Here’s how to diagnose and fix it in Prefect 3.x. First, a quick note on behavior - Global concurrency limits (GCL) track
active_slots
and may optionally use
slot_decay_per_second
(leaky-bucket/rate-limit semantics). If
slot_decay_per_second
> 0,
active_slots
will decay over time; if it’s 0,
active_slots
should drop immediately when leases are released or expire. - Leases auto-expire if not renewed (e.g., if a process crashes). A server service cleans up expired leases automatically. If you’re seeing capacity “shrink” over time, it’s usually stuck leases or a mismatch between your workload and
lease_duration
. How to confirm what’s happening 1) Inspect your limit(s)
Copy code
prefect gcl ls
prefect gcl inspect <LIMIT_NAME> --output json
Look for: - limit: configured capacity - active_slots: how many slots are currently in use (or accumulated for rate limits) - slot_decay_per_second: if non-zero, your usage decays over time (rate limiting) If
active_slots
stays high after your runs complete, you likely have stuck leases. If you’re still on the legacy (tag-based) limits:
Copy code
prefect concurrency-limit ls
prefect concurrency-limit inspect <TAG>
This view shows the task run IDs occupying slots. 2) Quick live check Run
prefect gcl inspect <LIMIT_NAME>
a few times over a few minutes. If no runs are active but
active_slots
doesn’t drop, leases may be stuck or the repossessor/cleanup isn’t running. 3) Common root causes - Leases not being released because the code didn’t use the
concurrency()
context manager. - Very long
lease_duration
combined with process termination or network interruptions. - Cleanup service not running (rare in Cloud; check server if self-hosted). - For rate-limit use cases, a high request rate with a small
slot_decay_per_second
can make it look like “capacity is shrinking” as
active_slots
accumulates faster than it decays. How to fix/reset - Global concurrency limits (3.x): - Reset active slots to zero:
Copy code
prefect gcl update <LIMIT_NAME> --active-slots 0
- Temporarily disable/enable:
Copy code
prefect gcl disable <LIMIT_NAME>
    prefect gcl enable <LIMIT_NAME>
- As a last resort, delete and recreate:
Copy code
prefect gcl delete <LIMIT_NAME>
    prefect gcl create <LIMIT_NAME> --limit <N> [--slot-decay-per-second <FLOAT>]
- Legacy tag-based limits:
Copy code
prefect concurrency-limit reset <TAG>
Best practices to avoid leaks - Always use the
concurrency()
context manager so leases are renewed and released automatically:
Copy code
from prefect.concurrency.asyncio import concurrency

  async with concurrency("my-limit", occupy=1):
      # your work
      ...
For sync code:
Copy code
from prefect.concurrency.sync import concurrency

  with concurrency("my-limit", occupy=1):
      ...
- Set an appropriate
lease_duration
for long-running work:
Copy code
with concurrency("my-limit", lease_duration=600):
      ...
- Consider
strict=True
if you want failures when renewals can’t be maintained. - If you’re self-hosting, ensure the server’s cleanup service is running so expired leases are reclaimed. If you can share a bit more, I can be more specific: - Are you on Prefect Cloud or self-hosted? - Prefect version? - Are you using global concurrency limits (
prefect gcl ...
) or tag-based? - The
prefect gcl inspect <LIMIT_NAME> --output json
result (especially
limit
,
active_slots
,
slot_decay_per_second
) and whether
active_slots
trends down when no runs are active.
m
@Marvin We actually have concurrency on our prefect workers which we set using --limit, -l The maximum number of flow runs to execute concurrently. How are these managed and kept track of and is there way to reset them over time.
m
thought for 300.7 seconds
Great question — the
--limit / -l
on
prefect worker start
is different from Prefect’s server-side concurrency limits. What
--limit
actually does - It’s a per-worker, in-memory cap on how many flow runs that single worker will execute concurrently. - It’s enforced inside the worker process (think “asyncio semaphore”), not by leases on the server. - It resets naturally as runs finish. There’s no decay/rate-limit behavior and nothing persisted on the server to “leak” over time. - To change it, you must restart the worker with a new
--limit
value. How it’s tracked - Tracked only within the worker process; the server does not keep a lease/slot count for a worker’s
--limit
. - If a worker crashes or is restarted, its concurrency state is reset immediately. If you suspect “stuck” worker slots - This typically means the worker believes it still has active runs (e.g., orphaned infra job, run never marked finished). - Easiest reset: restart the worker process. - Also check for lingering RUNNING flow runs and cancel them if needed:
Copy code
prefect flow-run ls --state RUNNING
  prefect flow-run cancel <FLOW_RUN_ID>
- Check worker logs for runs that never transitioned out of RUNNING. If you want a server-managed, adjustable cap (recommended) - Set a work pool–level concurrency limit. This is enforced across all workers in the pool and can be changed or cleared at runtime without restarting workers:
Copy code
prefect work-pool set-concurrency-limit "<POOL_NAME>" 10
  prefect work-pool inspect "<POOL_NAME>" --output json
  prefect work-pool clear-concurrency-limit "<POOL_NAME>"
- This is persisted on the server and won’t “shrink” over time. You can adjust it any time. If you want time-based “reset” or rate limiting - Worker
--limit
does not decay over time; it’s a hard concurrent run cap. - For rate-based behavior (e.g., “X per second/minute” that resets/decays), use a Global Concurrency Limit (GCL) with
slot_decay_per_second
and gate your flows with the
concurrency()
context. That’s separate from worker concurrency and gives you time-based control. Quick recommendations - Keep
--limit
on workers to bound per-process load. - Add a pool-level concurrency limit to globally cap throughput and adjust it dynamically without restarts. - If you’re seeing throughput “shrink,” restart the affected worker(s) and look for lingering RUNNING flow runs to cancel. A couple clarifying questions so I can tailor guidance: - Are you seeing a single worker stop picking up new runs even though none appear active? - Are you on Cloud or self-hosted? - Do you have a work pool concurrency limit set already, or are you relying solely on per-worker
--limit
?