A few things to check — in Prefect 2.x, Late usually means “the scheduler created the run, but no worker actually claimed it in time.” Given your description (pool ready, queue ready, workers ready), the most likely causes are: * a hidden pause somewhere * a stale concurrency setting * workers not actually polling/claiming work even though they look healthy * clock skew / API / DB issues A useful detail: in 2.x,

Late

is not terminal, but it often indicates the worker never successfully picked the run up. Things I’d check first 1. Confirm the work pool and queue are not paused Even if they show “ready”, a pause state can still block pickup. 2. Double-check concurrency on both the pool and queue Since you mentioned concurrency was set to

5

and then turned off yesterday, I’d be suspicious that something may still be stuck there. 3. Look at worker logs while a run is scheduled You want to see whether the worker is actually polling and whether it logs that it found/submitted a run. 4. Check server logs for scheduler / late-run / DB errors Sometimes the UI looks healthy while the API background services are throwing errors. 5. Check clocks on server vs worker hosts If they differ by more than a few seconds, Prefect can mark runs late unexpectedly. Important note on your specific symptom If new runs are all landing in

Late

, that strongly suggests the worker is not successfully claiming scheduled runs, rather than this being just leftover bad state from old runs. Concrete things to inspect Worker logs Look for errors around polling, API auth, connection failures, or submission failures. If you start a worker manually, you should see it actively polling:

bash
prefect worker start -p <your-pool-name>

Work queue health / last poll time If the queue’s

last_polled

is stale, the worker is not really connected even if it appears ready. Deployment paused state A paused deployment can still confuse diagnosis, so verify that too. Running/pending runs still counted against concurrency Even if you deleted old late runs, check whether some

RUNNING

/

PENDING

runs still exist and are consuming slots. If you want a quick isolation test Create a brand new deployment on a brand new work pool / queue with no concurrency limits at all. If that runs, the issue is likely stale config on the existing pool/queue rather than the worker host itself. What I’d try in order 1. Unpause/re-save the pool and queue explicitly 2. Remove any pool/queue concurrency limits again 3. Restart the worker and watch logs live 4. Schedule one test run 5. If it still goes

Late

, inspect Prefect server logs 6. If still unresolved, create a fresh pool + queue + deployment as a control test I didn’t find an exact matching GitHub issue/discussion for this symptom, but these docs may still help for 2.x concepts: * Prefect workers: Workers * Work pools / queues: Work pools If you want, paste: * one worker log snippet from startup through a run going

Late

* whether this is Prefect Server with Postgres or SQLite * whether the worker is on the same host as the API and I can help narrow it down faster.

Yep — that’ll do it. In practice for Prefect 2.x work pools: *

concurrency_limit = 0

→ no runs can be picked up *

concurrency_limit = null

→ no limit / unlimited So if the pool ended up at

0

, workers can stay healthy and queues can look ready, but every scheduled run just sits there and eventually becomes

Late

. Thanks for sharing the root cause — that’s a very useful gotcha. For anyone else hitting this: * if runs are stuck in

Late

* and workers / queues / pool all look healthy * check the work pool’s concurrency value carefully and make sure it’s not

0

If you want, I can also help turn this into a short GitHub discussion / community note since it seems easy for others to trip over.

Done — I created a discussion here: Work pool concurrency limit: `0` blocks all flow runs, while `null` means unlimited Thanks again for surfacing the root cause — that’s a subtle one, and it should help others debugging

Late

runs.