Got it — when many Prefect 2.x workers all poll the same queue, they can contend for claims and DB locks which slows draining. Here’s what reliably fixes it: Top fixes (in order of impact) 1) Use PostgreSQL for the API database - SQLite will lock under concurrent polling and drastically slow pickup. If you’re on SQLite, switching to Postgres is the single biggest improvement for multi-worker setups. - If you already use Postgres, skip this step. - Docs: Self-hosting Prefect (2.x) 2) Enforce concurrency at the pool and queue levels - Pool-wide cap so the DB only returns slots you can actually run:

prefect work-pool set-concurrency-limit <pool> <TOTAL_CONCURRENCY>

- Queue-level caps to shape work within the pool:

prefect work-queue create <queue-name> --pool <pool> --limit <QUEUE_LIMIT> --priority <PRIORITY>
# or, for an existing queue
prefect work-queue set-concurrency-limit <queue-name> <QUEUE_LIMIT> --pool <pool>

Docs: Work pools, Work queues, Concurrency limits 3) Shard across multiple queues (don’t have all workers hit a single queue) - Create N queues with their own limits and pin a subset of workers to each:

prefect work-queue create batch-a --pool <pool> --limit 10 --priority 5
prefect work-queue create batch-b --pool <pool> --limit 10 --priority 5

prefect worker start -p <pool> -q batch-a --limit 5 &
prefect worker start -p <pool> -q batch-b --limit 5 &

- This reduces query cardinality and prevents all workers from hammering the same queue. 4) Set a per-worker concurrency limit - Prevents a single worker from over-claiming and thrashing:

prefect worker start -p <pool> --limit <RUNS_PER_WORKER>

5) Lightly stagger worker polling (only if you still see contention) - Workers already include jitter, but with a lot of replicas you can further spread polls:

export PREFECT_WORKER_QUERY_SECONDS=15
export PREFECT_WORKER_HEARTBEAT_SECONDS=60
# Optional: claim earlier if infra spin-up is slow (Docker/K8s/etc.)
export PREFECT_WORKER_PREFETCH_SECONDS=20

prefect worker start -p <pool> -q <queue> --limit <RUNS_PER_WORKER>

Tradeoff: longer intervals = less DB churn but slightly slower pickup. Start around 10–15s. Quick diagnostic commands - Preview what each queue will hand out:

prefect work-queue preview <queue> --pool <pool>

- Inspect queue/pool limits:

prefect work-queue ls
prefect work-pool inspect <pool>
prefect concurrency-limit ls

Why this helps - DB-enforced pool/queue limits prevent over-claiming and reduce races. - Sharding queues means fewer workers competing on the same “hot” query. - Per-worker

--limit

keeps single workers from grabbing bursts and then idling while others wait. - Postgres removes SQLite’s write locks that cause long backoffs under load. If you can share a bit more, I can tailor exact numbers: - Are you on SQLite or Postgres? - How many workers, what worker type (process, Docker, K8s), and what are their

--limit

values? - Are all workers polling the same queue today? - Roughly how many queued runs and desired max concurrency? Happy to turn that into concrete pool/queue limits and env settings for your setup.

You’re right—sorry about the confusion. Let’s talk Prefect 3.x. What you’re seeing at ~1,000 runs with ~10 workers x 101 concurrency is not duplicate claiming (3.x uses atomic claims); it’s throughput bottlenecks from: - Lots of workers polling/claiming at once (queries + lock churn) - Concurrency slot math (pool/queue limits count PENDING + RUNNING), so prefetch can “fill” slots before work actually starts - DB characteristics (Postgres handles this well; SQLite will serialize under load) How claiming works in 3.x (why it’s still “contentious” at scale) - On Postgres, claims use row-level locks with FOR UPDATE SKIP LOCKED, so only one worker can actually claim a run. Others skip locked runs, but they still issue queries and compute available slots each poll. - Available slots are computed by counting PENDING and RUNNING runs per pool/queue on each poll. With 1,000 runs, that count + locking adds overhead. - If PREFECT_WORKER_PREFETCH_SECONDS is large, many runs sit in PENDING and consume slots before they start, which makes draining look slow. Recommendations to maximize drain rate 1) Make sure your pool limit matches your real target plus headroom - If you want ~10 x 101 ≈ 1,010 concurrent, set the pool to 1,100–1,200 to offset PENDING from prefetch, or reduce prefetch so PENDING doesn’t occupy as many slots.

prefect work-pool set-concurrency-limit my-pool 1200

2) Tune prefetch and polling for your infra - If your infra starts fast and you see lots of PENDING, lower prefetch so slots aren’t held too early:

export PREFECT_WORKER_PREFETCH_SECONDS=8
export PREFECT_WORKER_QUERY_SECONDS=8
prefect worker start -p my-pool --limit 101

- If your infra needs more runway to start (e.g., Docker/K8s image pulls), increase prefetch (e.g., 15–20s) but also raise the pool limit accordingly so PENDING doesn’t starve available slots. 3) Consider sharding across multiple queues - Split large backlogs across 2–4 queues and pin subsets of workers to each. This reduces per-query row scans and spreads lock attempts.

prefect work-queue create my-pool/part-a --limit 600
prefect work-queue create my-pool/part-b --limit 600
prefect worker start -p my-pool -q part-a --limit 101 &
prefect worker start -p my-pool -q part-b --limit 101 &

4) Keep per-worker limits explicit and consistent - You’re already using

--limit 101

which is good. Make sure all replicas have it, and size the pool/queues to the sum of worker capacity (plus prefetch headroom). 5) Check your backend - Prefect Cloud and self-hosted Postgres scale well here. If you’re on self-hosted SQLite, it will serialize writers under load and slow draining—migrating to Postgres is a big win. Useful commands to validate and monitor

# Verify pool/queue limits and slots
prefect work-pool inspect my-pool
prefect work-queue ls
prefect work-queue preview <queue> --pool my-pool

Docs for 3.x - Work pools: https://docs.prefect.io/v3/concepts/work-pools - Workers: https://docs.prefect.io/v3/concepts/workers - Settings & profiles (incl. PREFECT_WORKER_*): https://docs.prefect.io/v3/develop/settings-and-profiles If you can share: - Prefect Cloud vs self-hosted (and DB type) - Your current pool/queue concurrency limits - Current values for PREFECT_WORKER_QUERY_SECONDS and PREFETCH_SECONDS - Whether all workers poll the same queue or multiple queues …I can recommend exact numbers (pool limit, queue layout, and env settings) to get your drain time where you want it.