Short answer: because the work pool limit counts both Pending and Running. Your numbers show 72 Pending + 26 Running ≈ 98, which is essentially your pool limit of 100 in use. That’s why you only see ~26 Running at the moment. What those states mean in Prefect 3.x - Scheduled: queued for a worker to pick up; does not consume pool capacity. - Late: still Scheduled, but past its scheduled time; does not consume pool capacity. - Pending: a worker has claimed the run and is preparing/starting infrastructure; this DOES count toward the work pool limit. - Running: executing; also counts toward the work pool limit. Why many runs can be Pending - Infrastructure startup delays (e.g., container image pulls, Kubernetes scheduling, resource quotas) keep runs in Pending while consuming slots. - If your infra starts runs gradually, you’ll see more Pending than Running even though the pool limit is being honored. Quick checks - Inspect the pool to see limit and current usage:

prefect work-pool inspect "workpool-a"

- Confirm your work queue doesn’t have its own concurrency limit:

prefect work-queue ls
  prefect work-queue inspect <your-queue-name>

- Verify no deployment-level concurrency limits are set (you said none, but to double-check):

prefect deployment ls
  prefect deployment inspect "<flow-name>/<deployment-name>"

- Confirm workers weren’t started with a per-worker cap (e.g.,

--limit

in

prefect worker start

). If you want to increase “Running” specifically - Reduce Pending time by speeding up infrastructure startup (pre-pull images, scale cluster nodes, loosen quotas/limits). - Add worker capacity if a single machine is the bottleneck (CPU/IO bound). - If infra can handle it and you want more total in-flight runs, raise the work pool limit:

prefect work-pool set-concurrency-limit "workpool-a" 150

Note: this increases both Pending and Running capacity; if infra is slow, you’ll just see more Pending unless you address startup bottlenecks. If you share what infrastructure you’re using (Kubernetes, Docker, Process, etc.), I can suggest targeted tweaks to reduce Pending time.

Thanks — that helps. Two quick confirmations about concurrency in Prefect 3.x - It’s not only the work pool that can limit concurrency. - Work pool: global cap across all workers in the pool; counts both Pending + Running. - Work queue: can also have its own concurrency limit. - Deployment: may have an optional global concurrency limit (you said none, but worth double‑checking). - Verify with:

prefect work-pool inspect "workpool-a"
  prefect work-queue ls
  prefect work-queue inspect <your-queue-name>
  prefect deployment ls
  prefect deployment inspect "<flow-name>/<deployment-name>"

How to check if Kubernetes (EKS) startup is the bottleneck The symptom you described (many Pending, fewer Running) often means pods are slow to start. Here’s a focused checklist for EKS: 1) Compare Prefect timings - In the Prefect UI, open a flow run and view the timeline. If “Pending” duration is long before “Running,” infra startup is likely the bottleneck. - Turn on debug logs for the worker to see when jobs are submitted vs when they go Running:

PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p "workpool-a" -q "<queue-name>"

2) Inspect Kubernetes Jobs/Pods - List recent pods created by Prefect (labels typically include `prefect.io/flow-run-id`):

kubectl get pods -A -l '<http://prefect.io/flow-run-id|prefect.io/flow-run-id>' -o wide

- For a pod that sits Pending, describe it and check Events for scheduling/image pull issues:

kubectl describe pod <pod-name> -n <ns>
  kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -n 50

Common reasons: - Unschedulable: insufficient CPU/memory on nodes - ImagePullBackOff / ErrImagePull: slow or failing pulls from ECR - CreateContainerConfigError / permission issues - CrashLoopBackOff in the container 3) Check cluster capacity and autoscaling - See if you have many Pending pods cluster‑wide:

kubectl get pods -A --field-selector=status.phase=Pending

- Check node capacity/pressure:

kubectl get nodes
  kubectl describe node <node-name> | sed -n '/Allocated resources:/,/Events:/p'

- If using Cluster Autoscaler/Karpenter, confirm scale‑ups are occurring and how long they take (EKS console + autoscaler pod logs in kube-system). Long scale‑ups = long Pending. 4) Validate resource requests/limits in your Prefect K8s job template - If your base job template (on the work pool) requests large CPU/memory, pods can be unschedulable until nodes scale: - Reduce

resources.requests

to what you actually need. - Ensure

imagePullPolicy: IfNotPresent

when appropriate. - Check

nodeSelector

,

tolerations

,

affinity

aren’t over‑restrictive. - You can fetch/preview the pool template:

prefect work-pool get-default-base-job-template "kubernetes"
  prefect work-pool inspect "workpool-a"

5) ECR/image pull performance - Large images or cold nodes → long pulls. Mitigations: - Slim your image, or use a warm node group. - Pre‑pull images via a DaemonSet. - Ensure `imagePullSecrets`/IAM permissions are correct and fast (no repeated auth failures). 6) Namespace policies - Quotas/limits can throttle concurrency:

kubectl get resourcequota -n <ns>
  kubectl get limitrange -n <ns>

If you share: - the namespace you use, - a sample pod name that stayed Pending, - and whether you use Cluster Autoscaler or Karpenter, I can help read the events and suggest targeted fixes (e.g., tweak requests/limits, node group max size, image settings).

In Prefect 3.x, a run stays in Pending until the Kubernetes pod starts the flow runner and successfully tells the API “I’m Running.” If anything slows or prevents the pod from getting to that point, the run remains Pending. Most common reasons Pending doesn’t become Running (EKS/Kubernetes) - Pod not scheduled yet - Insufficient CPU/memory on nodes (Unschedulable) - Node selectors/affinity/tolerations too restrictive - ResourceQuota/LimitRange preventing scheduling - Cluster Autoscaler/Karpenter still scaling nodes up - Slow or failing image pulls - Large image size, cold nodes, ECR throttling - Missing/incorrect imagePullSecrets - ImagePullBackOff / ErrImagePull - Pod starts, but the flow runner can’t reach Prefect API - Missing/incorrect

PREFECT_API_URL

or

PREFECT_API_KEY

in the job env - Egress blocked (NACLs, security groups, proxies), DNS issues - TLS/proxy misconfiguration Result: pod runs, but it never sets the Prefect state to Running, so it stays Pending. - Pod starts, but the entrypoint never launches the runner - Custom job template overrides the command/args incorrectly - Wrong image that doesn’t contain

prefect

or the runner - CrashLoopBackOff before the runner initializes - Kubernetes backlog/limits - PDBs or admission webhooks delaying starts - Volume provisioning delays (PVCs/init containers) How to confirm which one it is 1) From Prefect - Open a Pending run in the UI and check the timeline. Long Pending before Running indicates infra startup issues. - Enable worker debug logs to see submission and job creation timing:

PREFECT_LOGGING_LEVEL=DEBUG prefect worker start -p "workpool-a" -q "<queue>"

2) From Kubernetes - Find pods for a specific Pending flow run (label includes flow-run-id):

kubectl get pods -A -l '<http://prefect.io/flow-run-id|prefect.io/flow-run-id>' -o wide

- If a pod exists but is Pending, inspect events for the reason:

kubectl describe pod <pod> -n <ns>
  kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -n 50

Look for Unschedulable, ImagePullBackOff, quota errors, etc. - If the pod is Running but Prefect still shows Pending, check logs for runner startup/API connectivity:

kubectl logs <pod> -n <ns> --all-containers --tail=200

You should see the runner starting and connecting to the Prefect API. If you see auth/connection errors, fix `PREFECT_API_URL`/`PREFECT_API_KEY` or network egress. 3) Cluster capacity and autoscaling - Pending pods cluster-wide:

kubectl get pods -A --field-selector=status.phase=Pending

- Node pressure/capacity:

kubectl get nodes
  kubectl describe node <node> | sed -n '/Allocated resources:/,/Events:/p'

- Check Cluster Autoscaler/Karpenter logs and timing in EKS; slow scale-ups = long Pending. 4) Job template sanity checks - Inspect the work pool’s base job template for resource requests/limits, node selectors, image, and command:

prefect work-pool inspect "workpool-a"

Make sure: - requests/limits match available capacity - image is correct and small enough - imagePullSecrets (if needed) are configured - command/args use Prefect’s runner (don’t override incorrectly) - no overly restrictive nodeSelector/affinity/tolerations Quick reminder on limits - Your pool limit counts Pending + Running. Seeing many Pending and fewer Running usually means infra startup is the bottleneck, not Prefect scheduling. If you share a specific Pending run ID and the pod name/namespace, I can help interpret the describe/logs and point to the exact cause.