Short answer: “Open spots” on a single work queue don’t guarantee runs will start. Runs go late when something else is gating them before they can move out of Scheduled. Most often it’s one of these: - Work pool concurrency is full: Even if your “critical” queue has a limit of 4, the parent work pool may have a global limit that’s already consumed by other queues/runs. - No worker is polling that queue: If no active worker is polling the “critical” queue, those runs won’t be picked up and will become Late. - Global concurrency limits are hit: If you’re using global concurrency keys/rate limits, those can block runs regardless of queue capacity. - Worker is polling multiple queues: The pool can be saturated by other queues the worker is polling, or priority across queues is causing your “critical” runs to be starved. - Queue/pool is paused or misconfigured: A paused queue or mismatch between queue and pool will keep runs from starting. - Infrastructure delays: Runs can sit in SUBMITTED/WAITING because your infra (K8s, Docker, etc.) can’t start them promptly — they’ll still be marked Late because they missed their scheduled start. What “Late” means: A run is marked Late when its scheduled start time has passed and it hasn’t begun transitioning to execution in time. It’s a symptom, not the root cause. Quick checks (Prefect 3.x): 1) Is the pool capped? - Check pool status and concurrency:

prefect work-pool inspect <your-pool>

If you see a pool concurrency limit that’s already in use, that’s the bottleneck. 2) Is the queue actually being polled and unpaused? - Inspect the queue:

prefect work-queue inspect critical -p <your-pool>
prefect work-queue preview critical -p <your-pool>

Preview shows what would be offered next and can reveal blocking conditions. 3) Is a worker polling this queue? - Make sure a worker is running and pointed at the pool and queue:

prefect worker start --pool <your-pool> --work-queue critical

Confirm in the UI that a worker is online for this pool and queue. 4) Any global concurrency limits? - In the UI, check the Concurrency page for any limits that might apply to these runs. Docs: Global concurrency limits 5) Infra constraints? - Look at a specific run’s timeline/states. If it’s stuck in SUBMITTED/WAITING, check your infra (K8s quotas, node capacity, Docker availability, etc.). Helpful docs: - Work pools: Concepts: Work pools - Work pool CLI: CLI: work-pool - Work queues CLI: CLI: work-queue - States (incl. Late): Concepts: States If you can share: - The work pool name and whether it has a concurrency limit - Whether a worker is running and polling the “critical” queue - Output of:

prefect work-pool inspect <your-pool>
prefect work-queue inspect critical -p <your-pool>
prefect work-queue preview critical -p <your-pool>

…I can pinpoint the exact cause.

A few easy ways to confirm a worker is actually polling your “critical” queue: UI - Go to Work Pools > your pool. - Check the Workers tab: make sure there’s an Online worker. Click it to see the list of work queues it’s polling — your “critical” queue should be listed. - Go to the Queues tab > select “critical”: you should see “Last polled” update regularly and (in many cases) which worker polled it. CLI - Inspect the queue:

prefect work-queue inspect critical -p <your-pool>

- Preview what the queue would offer if polled now (useful to spot blocking conditions):

prefect work-queue preview critical -p <your-pool>

Worker logs - On the machine where you started the worker, check its stdout/stderr. On startup it should print something like:

Polling work pool "<your-pool>" for flow runs from queues: ["critical"]

…and then periodic poll messages. If you don’t see “critical” listed, it isn’t polling that queue. Start (or restart) the worker with the right queue(s)

prefect worker start --pool <your-pool> --work-queue critical

Docs for reference: - Work pools: Concepts: Work pools - Work queue CLI: CLI: work-queue If you share screenshots or the output of:

prefect work-queue inspect critical -p <your-pool>
prefect work-queue preview critical -p <your-pool>

I can help confirm whether the queue is being polled and what might be blocking runs.

Good question — “recently polled” can be misleading. Two key points: - Last polled can be updated by “artificial” polls from the CLI/UI (e.g., work-queue preview/read-runs), not just by an online worker. So a recent poll does not guarantee a worker is actually running. - If no other queues have runs but your flows are still Late, it often means no worker is actively polling the queue or the pool can’t submit them. Do this to be sure: 1) Verify an online worker is polling the queue - In the UI: Work Pools > your pool > Workers tab. You should see an Online worker. Click it and confirm it lists your “critical” queue. - Or restart a worker explicitly on that queue:

prefect worker start --pool <your-pool> --work-queue critical

Then check its logs — you should see something like:

Polling work pool "<your-pool>" for flow runs from queues: ["critical"]

2) Check if the server would actually hand out runs from the queue - If preview shows runs waiting but your worker isn’t starting them, it’s a worker presence/capacity issue.

prefect work-queue preview critical -p <your-pool>

3) Confirm the runs you expect are in that queue and not filtered out - Open a late run in the UI and confirm its Work pool and Queue show your pool/“critical”. - Or list the runs on that queue (note: this triggers an artificial poll — don’t use it to infer a worker is online):

prefect work-queue read-runs critical -p <your-pool>

4) Rule out pool-level concurrency - A pool-wide limit can block the queue even if the queue’s own limit has room:

prefect work-pool inspect <your-pool>

5) Rule out global concurrency limits - In the UI, check Concurrency for any keys that might be at capacity. Docs: Global concurrency limits Helpful docs: - Work pools: Concepts: Work pools - Work queue CLI: CLI: work-queue - Work pool CLI: CLI: work-pool If you can share: - A screenshot of the pool’s Workers tab (showing Online workers) - Output of:

prefect work-pool inspect <your-pool>
prefect work-queue inspect critical -p <your-pool>
prefect work-queue preview critical -p <your-pool>

…I can pinpoint what’s blocking the runs.