Yes — it can be expected for runs to appear stuck, but indefinitely Pending is usually a signal to investigate, not normal steady-state behavior. The short version: * If runs are Pending, that usually means a worker has picked them up and is trying to submit/start them. * If runs are waiting on a deployment concurrency limit, they should more typically be in a scheduled waiting state such as

AwaitingConcurrencySlot

, not truly

Pending

. * A growing backlog of

Pending

runs often points more to worker/infrastructure submission bottlenecks or held concurrency slots than to raw machine CPU/RAM pressure alone. So: this may be a concurrency issue, but not necessarily because your box is out of resources. What to check 1. Work pool / worker capacity - If your workers can only submit a limited number of runs at once, runs may pile up before they reach

Running

. - This is separate from host CPU/RAM; you can have spare memory and still be bottlenecked by worker-side concurrency or submission behavior. 2. Deployment concurrency limits - If set, these can throttle how many flow runs from a deployment are allowed at once. - In Prefect 3, runs blocked by deployment concurrency are generally rescheduled to wait for a slot rather than sitting forever in

Pending

. 3. Work pool or queue concurrency limits - These can cap the number of

Pending

+

Running

runs pulled for execution. - If these are too low, backlog grows even if the host still looks healthy. 4. Stuck Pending submissions - A common pattern is that a run enters

Pending

, holds a slot, but never transitions to

Running

because infrastructure submission hangs or the worker dies/restarts. - In that case, downstream runs back up behind it. Why CPU/RAM alone doesn’t answer it 80% CPU and free RAM does not rule out concurrency misconfiguration. Suitable limits usually depend on: * how many flow runs a worker can reliably launch * external bottlenecks like DB connections, API rate limits, file handles, process startup overhead * whether flows are CPU-bound, IO-bound, or blocked on downstream systems * how long runs stay in

Pending

before entering

Running

Best practice for choosing a concurrency limit Start with an observed-capacity approach instead of picking a number from hardware specs alone: 1. Measure actual stable throughput - How many runs can execute concurrently without: - long

Pending

times - worker instability - external service saturation - increased failures/retries 2. Set limits below the failure threshold - If 20 concurrent runs is stable and 30 causes queueing/failures, use something like 15–20. 3. Use limits to protect bottlenecks, not just the worker host - Example: database, warehouse, API, or filesystem contention may be the real reason to cap concurrency. 4. Separate global and per-deployment concerns - Use deployment concurrency limits when one deployment is especially heavy or should not run too many copies simultaneously. - Use work pool / worker limits for overall infrastructure capacity. 5. Watch queueing behavior over time - Healthy system: runs may briefly wait, then move to

Running

. - Warning sign:

Pending

duration keeps increasing and never drains. Practical diagnosis steps I’d check these in order: 1. Look at run states distribution - Are the blocked runs really

Pending

, or are they

Scheduled

/

Late

/

AwaitingConcurrencySlot

? - If mostly

AwaitingConcurrencySlot

, that strongly suggests a concurrency limit. - If mostly

Pending

, that suggests worker/infrastructure submission issues. 2. Inspect deployment concurrency settings - Check whether affected deployments have concurrency limits configured. - If so, compare the limit to the number of currently active runs. 3. Inspect work pool / queue limits - Make sure the pool/queue isn’t capped lower than expected. 4. Check worker logs Look for patterns like:

- picked up flow run - submitting flow run - failure to start infrastructure - worker restart/crash - heartbeat interruptions 5. Check whether Pending runs ever become Running - If they never transition, the issue is likely not “we need more CPU” but “submission/startup path is hanging.” Rule of thumb If your workers are on-prem and the machine is healthy, but flow runs accumulate in

Pending

, I would suspect one of these before assuming host exhaustion: * worker concurrency too low * work pool / queue limit too low * deployment concurrency limit too low * some runs stuck in

Pending

and holding slots * infrastructure launch path hanging My recommendation For your case, I’d start by answering these three questions: 1. Are the blocked runs shown as

Pending

or

AwaitingConcurrencySlot

/

Scheduled

? 2. Do the affected deployments have deployment concurrency limits configured? 3. Are you using work pool or work queue concurrency limits in Cloud? If you want, I can help you narrow it down step-by-step if you share: * worker type (

process

,

docker

,

kubernetes

, etc.) * whether limits are set on the deployment, work pool, or queue * what state the “stuck” runs show in the UI * a snippet of worker logs around when a run gets stuck Relevant docs: * Deployments * Work pools * Workers * Global concurrency limits