Yes — in Prefect 2.20, a work pool concurrency limit of

10

does not guarantee you'll always observe 10 flow runs in the

Running

state at the same time. A few common reasons can explain why you only see

7–8

Running

while others remain `Pending`: 1. Work pool concurrency is not the only limiter Other limits may be in effect besides the pool limit: - work queue concurrency limits - deployment concurrency limits - tag-based concurrency limits - worker-side limits if the worker was started with

--limit

So even if the pool allows 10, something else may be reducing effective concurrency. 2.

Pending

runs can still be occupying pool capacity In Prefect 2.x, capacity accounting is often based on runs that have already been claimed/submitted by the worker, not only those visibly in

Running

. With Kubernetes workers, a run may spend time in

Pending

while Prefect has already reserved a slot for it: - worker has picked it up - job manifest is being created - k8s pod is being scheduled - image is pulling / container starting That means you can see: -

7–8

in

Running

-

2–3

in

Pending

- and still be at the pool limit of

10

So this is often expected behavior with k8s startup latency. 3. Kubernetes startup time creates a gap between submission and actual running This is especially common with k8s workers: - pod scheduling delay - image pulls - cluster autoscaling delay - namespace quota / resource pressure - unschedulable pods due to CPU/memory requests Prefect may have already submitted 10 runs, but only 7–8 pods are actually up and heartbeating yet. 4. Worker polling / prefetch behavior can make concurrency look underutilized Workers poll on an interval and prefetch upcoming work. Because of that, slot reuse is not always instantaneous: - one run finishes - slot is technically freed - worker has not polled again yet - next run remains pending briefly This can create small dips below the configured concurrency. 5. Multiple queues in the same pool can affect what gets picked up If the work pool has multiple work queues, priorities and queue-level behavior can make utilization look uneven. One queue may have available work that is not immediately being filled the way you expect. 6. Cluster-side limits, not Prefect, may be the real bottleneck Sometimes Prefect is fine, but Kubernetes is throttling actual execution: - node capacity exhausted - cluster autoscaler slow to react - pod affinity/taints/tolerations issues - resource quotas / limit ranges - job pods stuck in

Pending

In that case Prefect shows runs progressing, but only some become truly

Running

quickly. What I’d check first 1. Check whether the worker was started with its own limit If the worker has

--limit 8

, that would explain it immediately. 2. Check for queue/deployment/tag concurrency limits Especially tag-based and deployment-level concurrency. 3. Look at the k8s jobs/pods for the pending flow runs If the pods are still pending/scheduling/image-pulling, then this is probably k8s startup lag rather than Prefect refusing to submit more work. 4. Check worker logs at debug level Look for evidence that the worker is picking up 10 runs but some are taking time to transition. 5. Inspect whether multiple work queues exist in the pool Queue structure can matter. A few useful commands to run:

bash
prefect worker start --help
prefect work-pool inspect <pool-name>
prefect work-queue ls --pool <pool-name>
prefect concurrency-limit --help

If you want, I can help you narrow this down step by step if you share: - how you start the worker - whether you use multiple work queues - whether deployments/tags have concurrency limits - what state the k8s pods are in for those

Pending

runs Relevant docs: - Work pools and workers - Workers

- Concurrency limits One important note: those links are current Prefect docs, but your behavior in 2.20 is still broadly consistent with how workers + k8s submission behave.