Martynas Urbanavičius
03/04/2026, 9:43 AMMarvin
03/04/2026, 9:43 AMMarvin
03/04/2026, 9:48 AMactive_slots and may optionally use slot_decay_per_second (leaky-bucket/rate-limit semantics). If slot_decay_per_second > 0, active_slots will decay over time; if it’s 0, active_slots should drop immediately when leases are released or expire.
- Leases auto-expire if not renewed (e.g., if a process crashes). A server service cleans up expired leases automatically. If you’re seeing capacity “shrink” over time, it’s usually stuck leases or a mismatch between your workload and lease_duration.
How to confirm what’s happening
1) Inspect your limit(s)
prefect gcl ls
prefect gcl inspect <LIMIT_NAME> --output json
Look for:
- limit: configured capacity
- active_slots: how many slots are currently in use (or accumulated for rate limits)
- slot_decay_per_second: if non-zero, your usage decays over time (rate limiting)
If active_slots stays high after your runs complete, you likely have stuck leases.
If you’re still on the legacy (tag-based) limits:
prefect concurrency-limit ls
prefect concurrency-limit inspect <TAG>
This view shows the task run IDs occupying slots.
2) Quick live check
Run prefect gcl inspect <LIMIT_NAME> a few times over a few minutes. If no runs are active but active_slots doesn’t drop, leases may be stuck or the repossessor/cleanup isn’t running.
3) Common root causes
- Leases not being released because the code didn’t use the concurrency() context manager.
- Very long lease_duration combined with process termination or network interruptions.
- Cleanup service not running (rare in Cloud; check server if self-hosted).
- For rate-limit use cases, a high request rate with a small slot_decay_per_second can make it look like “capacity is shrinking” as active_slots accumulates faster than it decays.
How to fix/reset
- Global concurrency limits (3.x):
- Reset active slots to zero:
prefect gcl update <LIMIT_NAME> --active-slots 0
- Temporarily disable/enable:
prefect gcl disable <LIMIT_NAME>
prefect gcl enable <LIMIT_NAME>
- As a last resort, delete and recreate:
prefect gcl delete <LIMIT_NAME>
prefect gcl create <LIMIT_NAME> --limit <N> [--slot-decay-per-second <FLOAT>]
- Legacy tag-based limits:
prefect concurrency-limit reset <TAG>
Best practices to avoid leaks
- Always use the concurrency() context manager so leases are renewed and released automatically:
from prefect.concurrency.asyncio import concurrency
async with concurrency("my-limit", occupy=1):
# your work
...
For sync code:
from prefect.concurrency.sync import concurrency
with concurrency("my-limit", occupy=1):
...
- Set an appropriate lease_duration for long-running work:
with concurrency("my-limit", lease_duration=600):
...
- Consider strict=True if you want failures when renewals can’t be maintained.
- If you’re self-hosting, ensure the server’s cleanup service is running so expired leases are reclaimed.
If you can share a bit more, I can be more specific:
- Are you on Prefect Cloud or self-hosted?
- Prefect version?
- Are you using global concurrency limits (prefect gcl ...) or tag-based?
- The prefect gcl inspect <LIMIT_NAME> --output json result (especially limit, active_slots, slot_decay_per_second) and whether active_slots trends down when no runs are active.Martynas Urbanavičius
03/04/2026, 9:57 AMMarvin
03/04/2026, 9:57 AMMarvin
03/04/2026, 10:02 AM--limit / -l on prefect worker start is different from Prefect’s server-side concurrency limits.
What --limit actually does
- It’s a per-worker, in-memory cap on how many flow runs that single worker will execute concurrently.
- It’s enforced inside the worker process (think “asyncio semaphore”), not by leases on the server.
- It resets naturally as runs finish. There’s no decay/rate-limit behavior and nothing persisted on the server to “leak” over time.
- To change it, you must restart the worker with a new --limit value.
How it’s tracked
- Tracked only within the worker process; the server does not keep a lease/slot count for a worker’s --limit.
- If a worker crashes or is restarted, its concurrency state is reset immediately.
If you suspect “stuck” worker slots
- This typically means the worker believes it still has active runs (e.g., orphaned infra job, run never marked finished).
- Easiest reset: restart the worker process.
- Also check for lingering RUNNING flow runs and cancel them if needed:
prefect flow-run ls --state RUNNING
prefect flow-run cancel <FLOW_RUN_ID>
- Check worker logs for runs that never transitioned out of RUNNING.
If you want a server-managed, adjustable cap (recommended)
- Set a work pool–level concurrency limit. This is enforced across all workers in the pool and can be changed or cleared at runtime without restarting workers:
prefect work-pool set-concurrency-limit "<POOL_NAME>" 10
prefect work-pool inspect "<POOL_NAME>" --output json
prefect work-pool clear-concurrency-limit "<POOL_NAME>"
- This is persisted on the server and won’t “shrink” over time. You can adjust it any time.
If you want time-based “reset” or rate limiting
- Worker --limit does not decay over time; it’s a hard concurrent run cap.
- For rate-based behavior (e.g., “X per second/minute” that resets/decays), use a Global Concurrency Limit (GCL) with slot_decay_per_second and gate your flows with the concurrency() context. That’s separate from worker concurrency and gives you time-based control.
Quick recommendations
- Keep --limit on workers to bound per-process load.
- Add a pool-level concurrency limit to globally cap throughput and adjust it dynamically without restarts.
- If you’re seeing throughput “shrink,” restart the affected worker(s) and look for lingering RUNNING flow runs to cancel.
A couple clarifying questions so I can tailor guidance:
- Are you seeing a single worker stop picking up new runs even though none appear active?
- Are you on Cloud or self-hosted?
- Do you have a work pool concurrency limit set already, or are you relying solely on per-worker --limit?