@Marvin any thoughts on this?

I’m missing a bit of context—what are you referring to? If you can share a snippet or details, I’m happy to take a look. Helpful details: - What you’re trying to do (and expected vs. actual behavior) - Prefect version (2.x or 3.x) and how you’re running it (local, Cloud, work pool/worker type) - Deployment method (e.g.,

flow.from_source(...).deploy(...)

), storage, and infra details - Relevant code snippet and logs/error messages Drop a minimal example or a gist and I’ll dive in.

Thanks for the details — sorry you’re hitting this. What the errors mean - The engine is trying to acquire and release global concurrency limits via your API: - increment:

<http://48.211.249.12/api/concurrency_limits/increment>

- decrement:

<http://48.211.249.12/api/concurrency_limits/decrement>

- The name ‘audio’ in the error indicates your runs are attempting to acquire a concurrency limit with that name. The other string (

im-c0Ij4IOaaRYYCwnuAXqUmf

) looks like an internal/generated concurrency name. - A 500 from those endpoints means the server threw an exception while updating the concurrency counters; the client retries then gives up, causing the task to fail. Why this can happen even if you “never set any concurrency limits” - If a Global Concurrency Limit (GCL) exists for a tag name you’re using (e.g.,

audio

), Prefect will try to acquire a slot for that tag. If someone ever created a GCL named

audio

(even long ago), any run tagged

audio

will try to increment it. - If your code uses

prefect.concurrency

or

prefect.concurrency.rate_limit

, Prefect will attempt to acquire/release limits for the provided names. - Under spiky traffic, a misconfigured or stressed server/DB can 500 on concurrency increment/decrement calls, even if the name exists. First checks to run 1) List any existing concurrency limits (you may find

audio

or other unexpected limits):

prefect gcl ls
prefect gcl inspect audio

- If you see limits you don’t want, disable or delete them:

prefect gcl disable audio
# or
prefect gcl delete audio

Docs: Global concurrency limits, Tag-based concurrency limits 2) Confirm you aren’t using concurrency/rate limits in code - Search for

prefect.concurrency

or

prefect.concurrency.rate_limit

and remove/adjust if you don’t intend to enforce limits. 3) Verify your server and database setup - Prefect 3.x self-hosted requires PostgreSQL 14.9+ (SQLite is dev-only and will struggle under bursty traffic). See: Self-hosted docs and recent note: “Document PostgreSQL 14.9+ requirement.” - Make sure your DB migrations are up to date:

prefect server database upgrade

- Check the API server logs around the time of the 500s for a stack trace from the concurrency endpoints; that will pinpoint whether it’s a DB lock/constraint issue, a migration mismatch, or something else. 4) If you do want concurrency but are seeing blocking during spikes - Consider increasing the limit on the relevant GCL or temporarily disabling it to recover:

prefect gcl update audio --limit <higher_number>
# or disable to unblock
prefect gcl disable audio

- If you’re rate-limiting, tune

slot_decay_per_second

to smooth bursts: Rate limiting with slot decay A few targeted questions to help me narrow this down - Prefect version (e.g., 3.4.x) and how you deploy the server (single API replica? behind a load balancer?). - Database type and version (PostgreSQL version or SQLite?). - Output of:

prefect gcl ls
prefect gcl inspect audio

- Any use of

tags=["audio"]

on flows/tasks or

with tags("audio")

? Any use of

prefect.concurrency

or

prefect.concurrency.rate_limit

in your code? - A snippet of the server logs around the 500 errors from

/api/concurrency_limits/increment

or

/decrement

. Why tasks are erroring - The concurrency acquisition service retries but if the server keeps returning 500, it raises

ConcurrencySlotAcquisitionError

and the task run fails. Fixing the underlying 500 (DB/migration/load) or removing the limit causing acquisition should immediately stop the failures. References

- Global concurrency limits: Guide - Tag-based concurrency: Guide - REST API (increment/decrement): Increment, Decrement - Latest release notes: GitHub releases (recent changes around concurrency/leases) If you can share the version, DB details, and

gcl

outputs, I can help you get to a concrete fix quickly.