Short answer: those runs are going “Late” because they’re sitting in Scheduled and aren’t being picked up fast enough by a worker, then they start only as capacity frees up. The “one‑by‑one” clearing means there’s an effective bottleneck allowing just a trickle of new starts. What “Late” means in Prefect - Late is just a Scheduled variant the server applies when a run’s scheduled start time passes by ~15s without starting. - It does not cancel anything—it’s a signal that nothing picked the run up in time. Why this happens in your scenario Given: - 100 runs currently executing - You trigger 200 more - You “have 180 runs concurrency limit” (not sure where this is configured) Likely causes: - Worker/process limit: Your Kubernetes worker(s) are only allowed to start N runs at once (e.g.,

prefect worker start --limit N

). If N≈100, the next 200 remain Scheduled → Late; they’re picked up only as slots free. - Work queue/work pool limit: A concurrency limit on the queue/pool may cap the number of runs that can be fetched. - Deployment concurrency limit: If set lower than needed, extra runs are queued (“AwaitingConcurrencySlot”). Note: if deployment limits were blocking, you’d typically see “AwaitingConcurrencySlot” instead of “Late.” - Kubernetes capacity/quotas: If the cluster doesn’t have ≥180 CPUs/GB free, k8s will delay scheduling pods. Prefect runs linger in Scheduled and get marked Late until pods can actually start. This often looks like “one new pod starts when one finishes.” - Image pull or API throttling: Slow image pulls or API rate limits can make worker submissions slow enough that many runs go Late, then start in a slow trickle. How to pinpoint the bottleneck quickly Prefect side: - Check the run state names for the waiting runs: - “Late” = no worker picked them up in time. - “AwaitingConcurrencySlot” = blocked by a Prefect concurrency limit (usually deployment-level). - Check deployment(s): is a

concurrency_limit

set, and what is the collision strategy? - Check work pool/queue: any concurrency limit configured? - Check workers: - How many Kubernetes workers are running? - What

--limit

is each worker using? Combined limit should be ≥ your target concurrent runs. Kubernetes side: - Are new pods Pending?

kubectl get pods

and

kubectl describe pod

to see events like insufficient CPU/Memory, quota, or taints. - Check namespace ResourceQuotas/LimitRanges. - If using a cluster autoscaler, check scale-up latency. - Check image pull times and registry throttling. Remediations - Match capacity to demand: - Run enough Kubernetes workers and set

--limit

high enough to cover your desired concurrency (e.g., total ≥180). - Ensure deployment/work queue/pool concurrency limits (if any) are not lower than your target. - Verify the cluster actually has resources for 180 pods at 1 CPU/1 GiB each (and room for system overhead). - If Late is just noise (expected backlog), increase the Late threshold on the server so runs don’t get marked Late so quickly: - Set

PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS

(e.g., 300) on your Prefect API/Server. - If image pulls are the bottleneck, use a node-local registry cache, pre-pull images, or smaller images. A few clarifying questions to help us narrow it down: 1) Where is the “180 runs concurrency limit” set—deployment, work queue/pool, or worker

--limit

? 2) How many Kubernetes workers are you running, and what is each worker’s

--limit

? 3) In the UI, do waiting runs show “Late” or “AwaitingConcurrencySlot”? 4) Do you see many pods Pending in Kubernetes when this happens? If so, what do

kubectl describe

events say?

Thanks — that helps. Given your setup: - 6 work pools, each with a pool-level limit of 30 - 2 workers per pool (12 workers total) - New runs show “Late” (not “AwaitingConcurrencySlot”) What’s happening - “Late” means runs stayed in Scheduled past their scheduled time because no worker claimed them fast enough. It does not mean they’re blocked by a Prefect concurrency limit on the deployment (that would show “AwaitingConcurrencySlot”). - With a pool limit of 30, each pool will only allow ~30 active runs (Pending/Running/Cancelling) at a time. If you burst 200 additional runs into one or a subset of pools that are already near 30, the extras will remain Scheduled and quickly flip to Late. They’ll start “one-by-one” as each finished run frees a pool slot. - This also happens if a pool has fewer/slow workers: unclaimed runs stay Scheduled → Late until a worker fetches them. Why you only see ~100 running when the “total” could be 180 - The 180 is the sum of per-pool caps (6 × 30). You only reach 180 if runs are distributed roughly evenly across all 6 pools and each pool has healthy workers. - If your 200-run burst mostly targets 3–4 pools, those pools will cap at 30 and the rest of the runs in those pools will sit as Late. Meanwhile, the other pools might be idle or underutilized. How to confirm quickly - In the UI, filter the affected runs by Work Pool. Check the “active” count per pool vs that pool’s limit=30. - Click into a Late run and confirm it’s assigned to a pool that’s already at its cap. - Check worker logs for each pool to make sure both workers are running and claiming runs. If you started workers with a

--limit

, that can further cap each worker’s local concurrency. - If your Kubernetes cluster were the bottleneck, you’d typically see runs progress to Pending/Running while pods sit Pending at the cluster level. Since you’re seeing Late, the bottleneck is before K8s scheduling (i.e., the runs aren’t even being claimed). Options to fix or improve - Balance or increase per-pool capacity: - Spread runs evenly across all six pools (e.g., via multiple deployments pointed at different pools). - Raise the concurrency limit on the specific pool(s) taking the burst (if your cluster can handle it). - Ensure both workers per pool are healthy and not started with a very small

--limit

. - If “Late” is just expected during bursts and you want less noise, increase the lateness threshold on the Prefect server:

# On your Prefect API/Server
  export PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS=300

That means runs won’t be marked Late until 5 minutes after their scheduled time. - If claim speed is the issue (not the cap), add more workers to the hot pools or reduce worker poll intervals so they claim faster. But the pool limit will still enforce the cap of 30 per pool. A couple of quick questions to pinpoint it: - Are the 200 new runs mostly going to the same 1–2 pools? - Did you set any

--limit

on the workers? - In the UI, do those pools show ~30 active runs when the backlog is Late? If yes, that’s the pool cap doing its job. If you can share one Late run’s deployment and pool and a screenshot of that pool’s active count, I can confirm and suggest the exact adjustment.