Thanks for the details — this pattern usually points to a coordination issue in self-hosted, multi-API setups rather than a “truly full” limit. High-probability root causes in GKE with multiple API pods - Non-shared concurrency lease storage across API replicas. By default Prefect Server uses an in-memory lease store; with multiple API pods, each pod tracks leases separately. That can cause slot acquisition to fail intermittently even when the UI shows free slots. - Docket not backed by Redis. With multiple server components, the scheduler/automations/concurrency coordination need a shared Docket backend (Redis). Using the default in-memory Docket in a multi-pod setup leads to inconsistent behavior. - Aggressive retry/backoff hitting a timeout in your code. If you use the

concurrency()

context manager with `timeout_seconds`/`max_retries` (and/or

strict=True

), transient 423 responses under contention can bubble up as ConcurrencySlotAcquisitionError even though the limit is “mostly free”. - Long-running tasks with short lease durations leading to churn/renewal failures under load. Quick triage to confirm it’s multi-replica related - Temporarily scale your API deployment down to 1 replica and re-run. If the errors disappear, it’s almost certainly lease storage/Docket configuration across replicas. What to verify and adjust 1) Are you using

concurrency()

in code or only server-side tag-based gating? - If you’re using

concurrency()

with tag names (e.g.,

concurrency("tag:gpu")

), please share a quick snippet including

strict

,

timeout_seconds

, and

max_retries

. This error comes directly from the context manager when acquisition times out or strict mode fails. - If you are only tagging tasks (

@task(tags=[...])

) and relying on server-side tag concurrency, a flow-level ConcurrencySlotAcquisitionError typically means your code also uses

concurrency()

somewhere. 2) Configure a shared lease storage for concurrency across API pods - Install the Redis integration and point Prefect Server to the Redis lease storage:

pip install prefect-redis
# or add it to your image

- Set these environment variables on all Prefect Server API pods:

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE=prefect_redis.lease_storage.ConcurrencyLeaseStorage
PREFECT_REDIS_MESSAGING_HOST=<your-redis-host>
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# optional if secured
PREFECT_REDIS_MESSAGING_USERNAME=<user>
PREFECT_REDIS_MESSAGING_PASSWORD=<pass>
PREFECT_REDIS_MESSAGING_SSL=true

Docs: - Prefect-Redis lease storage API - Scaling self-hosted Prefect 3) Ensure Docket uses Redis (not in-memory) when running multiple pods - Set on all server components (API, scheduler, automations):

PREFECT_SERVER_DOCKET_URL=redis://<your-redis-host>:6379/0

Docs: - Self-hosted scaling (scheduler/docket) 4) Keep a single background scheduler - You said you’re already doing this — great. Still worth confirming only one scheduler pod is running. 5) Tune backoff to reduce thundering herd during brief contention - Increase the server’s max wait for tag-based limits:

PREFECT_SERVER_TASKS_TAG_CONCURRENCY_SLOT_WAIT_SECONDS=30

- If using

concurrency()

in code, increase

timeout_seconds

and/or

max_retries

, and avoid

strict=True

unless you truly need hard-fail behavior:

from prefect.concurrency.asyncio import concurrency

async with concurrency(
    "tag:my-tag",
    timeout_seconds=120,   # longer window to acquire
    max_retries=10,        # more retries under load
    lease_duration=600,    # if tasks are long-running
    strict=False           # don’t hard-fail if the limit is missing
):
    ...

Reference: - How-to: tag-based concurrency limits

- API: concurrency (async) - API: concurrency (sync) - Concepts: tag-based limits - Concepts: global limits 6) Diagnostics you can run now - Server logs (API pods) — enable concurrency logs to see 423 and retry details:

PREFECT_LOGGING_LEVEL=DEBUG
PREFECT_LOGGING_LOGGERS='["prefect.server.api.concurrency_limits_v2","prefect.server.orchestration","prefect.concurrency"]'

- Check DB state for the relevant limits (Postgres):

SELECT name, active_slots, limit, denied_slots, slot_decay_per_second, updated
FROM concurrency_limit_v2
WHERE name IN ('tag:your-tag') AND active = true;

- If you switch to Redis lease storage, you can also inspect leases:

redis-cli ZRANGE "prefect:concurrency:expirations" 0 -1 WITHSCORES

A couple clarifying questions to help pinpoint the fix - Which Prefect version are you on? - Are you using

concurrency()

in your flow/task code with a tag name (e.g.,

concurrency("tag:...")

)? If yes, can you share the arguments you pass (strict/timeout/max_retries/lease_duration)? - Have you configured

PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE

to use Redis, or are you only using Redis for events/messaging? - Can you try scaling the API deployment to a single replica briefly to see if the error disappears? If you share a snippet of how you’re applying the concurrency limit and your server env configuration, I can suggest exact settings for your GKE deployment.