If a deployment run is going Late, the main thing to know is: A scheduled run was created, but no worker picked it up in time. So the places to check are usually: 1. Work pool / worker 2. Work queue 3. Concurrency limits 4. Deployment schedule 5. Worker logs / run logs A run is marked Late when it stays in

Scheduled

past its scheduled start time for too long. In Prefect, that usually means the worker never submitted it for execution. Where to look first 1) Work Pool page in Prefect UI Go to your Work Pool and check: - Is there an active worker attached? - Does it show a recent Last Polled time? - Is the worker polling the correct pool/queue? If the worker is not polling, runs will sit in

Scheduled

and then become

Late

. Docs: - Work pools - Deployments 2) Work Queue status Inside the work pool, check the queue: - Is the queue paused? - Is it Ready / Not Ready? - Does the deployment actually target that queue? - Is there a queue-level concurrency limit already full? If the queue is paused or full, scheduled runs won’t start. 3) Deployment page Open the deployment and verify: - Schedule is active - It points to the expected work pool / queue - New scheduled runs are actually being created - The Late runs belong to the deployment you expect 4) Flow run details Click one of the Late runs and inspect: - State history - Logs - Infrastructure/job variables - Any events around submission If there are no logs at all, that often means the worker never picked up the run. Common root causes Worker is not actually polling Even if it looks “healthy”, it may not be polling the right pool/queue. Run your worker with debug logs and watch for polling/submission messages:

bash
prefect worker start --pool <your-pool-name> --log-level DEBUG

You want to see logs indicating it is querying for scheduled runs and submitting them. Queue or pool mismatch Sometimes the deployment is assigned to one work pool/queue, but the worker is listening on another. Check: - deployment’s work pool - worker’s pool - queue filters if you use them Concurrency limit is blocking runs Check both: - work queue concurrency limit - global concurrency limits - deployment/tag-related limits if configured If all slots are used, new runs can stay scheduled and then become Late. Docs: - Global concurrency limits Infrastructure submission failure The worker may see the run but fail when trying to start infrastructure: - Docker image pull issue - Kubernetes job creation failure - missing credentials - bad job variables - network permissions In that case, worker logs are the best place to inspect. Schedule exists, but worker misses the pickup window Late is often just a downstream symptom of delayed polling, worker restarts, or blocked infrastructure. Best troubleshooting checklist 1. Open the Late flow run - Check state history - Check whether any logs exist 2. Open the deployment - Confirm schedule is active - Confirm correct work pool/queue 3. Open the work pool / queue - Confirm worker is attached - Confirm queue is not paused - Confirm queue is ready - Confirm concurrency is not exhausted 4. Check worker logs with

DEBUG

- Is it polling? - Is it finding runs? - Is it trying to submit them? - Is submission failing? 5. Check infra-specific logs - Docker / Kubernetes / VM / ECS / ACI depending on worker type Very likely explanation in your case Since you said: - deployments are healthy - work pools are healthy - but runs become Late the most likely causes are: - worker is polling the wrong pool/queue - queue is paused or filtered unexpectedly - concurrency limit is full - worker sees the run but cannot submit infrastructure - worker polling is delayed/stuck Useful docs

- Work pools - Deployments - Schedules - States - Concurrency limits If you want, I can help you debug this step by step. If you send: - your worker type (

process

,

docker

,

kubernetes

, etc.) - how you start the worker - whether this is Prefect Cloud or self-hosted - a screenshot or copy of one Late run’s state/logs I can narrow down the exact place to check next.

@Marvin in prefect Version 3.6.0 in our dev instance there are so many flows showing status as Late and for some flows it works how to debug and fix the issue ?

Yes — in Prefect

3.6.0

, if many runs are becoming

Late

but some still work, that usually means: the scheduler is creating runs, but some workers/queues/infrastructure are not picking them up fast enough. So this is usually not a flow code problem — it is usually one of these: - worker not polling correctly - wrong work pool / wrong queue - paused or unhealthy queue - queue/deployment/global concurrency limit reached - worker can see runs but cannot submit infrastructure - time skew / server load in dev environment Best way to debug Start with one Late run and trace it backward: 1. Open a Late flow run in the UI Check: - state history - logs - deployment name - work pool / work queue - scheduled start time If there are no worker logs at all for that run, the worker likely never picked it up. 2. Open the deployment for that run Verify: - schedule is active - deployment points to the correct work pool - queue is what you expect - deployment-level concurrency is not blocking it 3. Open the work pool and queue Check: - queue is not paused - queue is

Ready

- Last Polled is recent - worker count is non-zero - concurrency is not exhausted If Last Polled is old, the worker is your first suspect. 4. Check the worker process logs Run worker with debug logging if possible:

bash
prefect worker start --pool <your-pool-name> --log-level DEBUG

Look for messages like: - querying for scheduled runs - discovered X runs - submitting flow run - failed to submit infrastructure If the worker keeps polling but does not pick certain runs, then check queue filters and concurrency. Most common causes when only some flows are Late 1) Concurrency limits This is very common. Check: - work queue concurrency limit - deployment concurrency limit - global concurrency limits If some flows run and others stay Late, often one queue is full while others are not. Docs: - Global concurrency limits 2) Wrong queue / pool assignment Some deployments may be attached to a different queue than the worker is polling. Double-check: - deployment work pool - deployment queue - worker pool - worker queue selection 3) Queue paused or not ready A queue can exist but not actually be serviced. 4) Worker can poll but cannot start infrastructure If using Docker/Kubernetes/etc., the worker may find the run but fail to launch it. Examples: - bad image - registry auth issue - k8s job creation failure - permissions issue - resource shortage in dev cluster 5) Dev environment performance / timing By default, runs can be marked Late pretty quickly. In a slower dev environment, workers may not pick runs up before the Late threshold. That can happen if: - server is underpowered - worker host is overloaded - database is slow - API response is slow What I would check in order Quick checklist - [ ] Pick one Late run and inspect its logs/state history - [ ] Confirm the deployment’s work pool and queue - [ ] Confirm the queue is not paused - [ ] Confirm queue

Last Polled

is recent - [ ] Confirm worker is attached to that exact pool/queue - [ ] Check queue/deployment/global concurrency limits - [ ] Check worker debug logs for submission failures - [ ] Check infrastructure logs if using Docker/Kubernetes - [ ] Check machine clock/time sync on server + worker - [ ] Check whether this happens only for specific deployments Important note for self-hosted dev instances If this is self-hosted Prefect Server, also check: - server resources - API responsiveness - database performance - whether orchestration/scheduler services are healthy If this is Prefect Cloud, then focus more on: - workers - queues - deployment config - infra submission Useful commands Start worker with debug logs:

bash
prefect worker start --pool <pool-name> --log-level DEBUG

That command is valid in Prefect 3.x. How to fix once you find the cause

- If worker is not polling -> restart/fix worker - If queue is paused -> unpause queue - If wrong pool/queue -> update deployment or worker config - If concurrency full -> raise limit or reduce stuck/running jobs - If infra launch failing -> fix Docker/K8s/image/credentials/resources - If dev instance is slow -> scale server/DB/worker resources, or relax Late sensitivity if appropriate Docs - Work pools - Deployments - Schedules - States - Concurrency limits If you want, I can help you narrow this down quickly. Please send: - whether this is Prefect Cloud or self-hosted server - worker type:

process

,

docker

,

kubernetes

, etc. - one example deployment that works - one example deployment that becomes Late - the worker startup command - screenshot or text from one Late run’s state/logs With that, I can tell you the most likely root cause and next exact check.

Yes — here’s how to think about it. 1) Where to inspect logs in Prefect UI For a specific Late run: - Go to Deployments or Flow Runs - Click the specific flow run - Open the Logs tab If you do not see any logs there, that usually means: - the worker never picked up the run, or - the run never actually started execution In that case, the most useful logs are often worker logs, not flow logs. Also check in the run details page: - State history / timeline - deployment name - work pool / queue - scheduled time If the run only shows

Scheduled -> Late

and no execution logs, the worker likely never launched it. 2) Does

Late

count toward concurrency? Usually, Late runs are still scheduled runs, not running runs. But for work queue concurrency, Prefect can count both

Pending

and

Running

toward active slots, depending on queue mechanics — not just

Running

. A

Late

run itself is not the same as

Running

. So practically: -

Running

definitely consumes concurrency -

Pending

can also matter for queue capacity -

Late

means it was not picked up in time -

Late

is not usually “executing” So if you have many Late runs, the issue is usually not that those Late runs are actively consuming compute, but that something is preventing new scheduled runs from being submitted. 3) How to debug when some flows in other work pools are still running That’s actually a very helpful clue. If other work pools are running fine, then your Prefect server/cloud is probably okay overall. That points more toward a problem specific to: - one work pool - one work queue - one worker - one deployment configuration - one infrastructure type So compare a working pool vs a Late pool: Check these side by side: - work pool name - work queue name - queue paused or not - queue last polled time - worker count - worker type - concurrency limits - infrastructure type (

process

,

docker

,

kubernetes

, etc.) - deployment job variables 4) Best debug path For one Late run: - open the run - check Logs - check State history If no logs: - go to the deployment - see which work pool and queue it uses - go to that work pool - inspect queue health and Last Polled Then compare to a working one. 5) What to inspect in worker logs Start or review the worker logs for the affected pool:

bash
prefect worker start --pool <pool-name> --log-level DEBUG

Look for: - polling for scheduled runs - number of runs discovered - submitting a flow run - errors launching infrastructure - queue/concurrency related messages If worker logs show: - no runs discovered → queue/filter/deployment mismatch - run discovered but not submitted → concurrency or orchestration issue - submission failed → infrastructure problem 6) Simple answer on concurrency If you’re asking: “Do lots of Late runs block concurrency the same way running flows do?” Short answer: No, not like Running flows do. But the queue may already be saturated by: -

Running

runs -

Pending

runs - queue/deployment/global limits So Late runs are usually more of a symptom than the thing consuming the slots. 7) What I recommend you check right now For one affected work pool: - open work pool page - check queue status - check

Last Polled

- check queue concurrency limit - count currently

Running

/

Pending

runs - inspect worker logs Then compare with a healthy work pool that is still running flows. If you want, I can give you a very specific UI checklist like: “click here -> then here -> what value to look for” for Prefect UI, or help you interpret one screenshot from: - a Late run page - the work pool page - the worker logs

Yes — that message is very important. If the run says: “This run is scheduled and hasn't generated logs” that usually means: the flow never actually started execution. So the problem is before your flow code runs. That means you should focus on: - worker pickup - work pool / queue config - concurrency limits - infrastructure submission - deployment targeting the wrong queue/pool Not on the flow code itself. What to compare: working work pool vs non-working work pool Compare these side by side in the UI. 1. Work pool details For both the working and non-working pool, check: - pool name - pool type (

process

,

docker

,

kubernetes

, etc.) - paused or not - worker count - base job template / infrastructure settings Questions: - Is the broken pool paused? - Is it the same worker type as the working pool? - Did someone change its infrastructure config recently? 2. Work queue details Inside each pool, compare the queue: - queue name - queue paused or not - queue status:

Ready

vs

Not Ready

- Last Polled timestamp - concurrency limit - whether the deployment is routed to this queue Most important: - if the working queue has a recent Last Polled time and the broken one does not, then the worker for that pool/queue is not polling correctly - if the broken queue is paused, that is likely the reason - if the broken queue concurrency is full, runs may not start 3. Worker side Compare the actual worker processes: - is a worker process running for the broken pool? - is it started with the right

--pool

? - if using queue selection, is it started with the right queue? - are there any worker errors in logs? Example:

bash
prefect worker start --pool <pool-name> --log-level DEBUG

If the worker is attached to the wrong pool, Late runs will happen even if the pool exists in the UI. 4. Deployment config Pick one deployment from the broken pool and one from the working pool and compare: - work pool - work queue - schedule active/inactive - concurrency limit - job variables / infrastructure overrides A common issue is: - deployment says it uses pool A - worker is polling pool B 5. Running / pending load Check whether the broken pool already has many: -

Running

-

Pending

Even if the Late run itself is not running, the pool/queue may already be full. What the “no logs” case strongly suggests If there are no logs, then usually one of these happened: Case 1: No worker picked up the run Most common. Look for: - old Last Polled - no active worker on that pool - wrong queue/pool assignment Case 2: Worker picked it up too late or failed before flow execution Look in worker logs for: - infrastructure creation failed - permission error - image pull error - Kubernetes job creation failure - Docker daemon issue Case 3: Concurrency is blocking submission Look for: - queue concurrency limit reached - deployment concurrency limit reached - global concurrency limit reached What to inspect first in the UI For the non-working work pool, look for these exact things: In Work Pool page - Is the pool paused? - Is there a worker listed? - Is worker heartbeat/recent activity visible? In Queue page - Is queue paused? - Is queue

Ready

? - What is Last Polled? - Is concurrency limit set? - Are there many Late runs associated with it? In one deployment - Is schedule active? - Is deployment assigned to this exact pool/queue? Practical compare table Use something like this:

text
Working pool:
- Pool paused? No
- Queue paused? No
- Queue status? Ready
- Last polled? 10 seconds ago
- Worker running? Yes
- Concurrency full? No
- Deployments starting? Yes

Non-working pool:
- Pool paused? ?
- Queue paused? ?
- Queue status? ?
- Last polled? ?
- Worker running? ?
- Concurrency full? ?
- Deployments starting? No

If you fill this out, the difference usually becomes obvious. Biggest signal to watch

If working pool has a recent Last Polled and non-working pool does not, then the issue is almost certainly: the worker for the non-working pool is not polling, is stuck, or is pointed at the wrong pool/queue. Next best step Please check these 4 values for the non-working pool and send them: - pool type - queue status - queue last polled - worker startup command And if possible, the same 4 values from a working pool. Then I can help you pinpoint the likely root cause very quickly.

Yes — let’s try to get you to a fix today. Since the Late runs show no logs, the fastest path is: assume the worker/queue/pool path is broken until proven otherwise. Here’s the shortest fix-first checklist. Immediate recovery steps 1. Identify one broken deployment Pick one deployment that always goes

Late

. From its page, note: - work pool name - work queue name - worker type (

process

,

docker

,

kubernetes

, etc.) 2. Check the queue in the UI Go to the deployment’s work pool and verify: - queue is not paused - pool is not paused - queue status is Ready - Last Polled is recent If Last Polled is stale or missing: - the worker is not polling that queue correctly 3. Restart the worker for that pool Use the worker attached to the broken pool and restart it. Valid Prefect 3.x command:

bash
prefect worker start --pool <pool-name> --log-level DEBUG

If the worker is already running, stop it and restart it cleanly. Then watch for: - polling messages - discovered scheduled runs - submitting flow run - infrastructure errors 4. Trigger one manual run After restarting the worker, manually trigger one run for the broken deployment. Watch whether it: - moves to

Pending

/

Running

- stays

Scheduled

- becomes

Late

again That tells us where it fails. 5. If it still does not start, compare with a working pool Compare broken vs working: - same worker type? - same pool type? - same queue status? - same last polled behavior? - same infra settings? - same concurrency limits? Most likely fixes Fix A: Worker is attached to wrong pool/queue Restart worker with the correct pool.

bash
prefect worker start --pool <correct-pool-name> --log-level DEBUG

If you use queue selection, make sure it matches the deployment queue too. Fix B: Queue or pool is paused Unpause it in the UI. Fix C: Concurrency limit is full Check: - work queue concurrency - deployment concurrency - global concurrency If full, either: - raise the limit - cancel/finish stuck runs - move workload to another queue Fix D: Infrastructure launch is failing If worker logs show Docker/Kubernetes/etc. errors, fix those: - image exists - credentials are valid - cluster reachable - resources available Fix E: Dev worker is stale or hung A simple worker restart often fixes this in dev. If you need the fastest practical route today Do this in order:

text
1. Pick one broken deployment
2. Confirm its pool/queue
3. Confirm queue is not paused
4. Check Last Polled
5. Restart worker for that pool with DEBUG logs
6. Trigger one manual run
7. Watch worker logs

If there are still no flow logs after manual trigger, the issue is definitely before flow execution. What I need from you right now to help fix it quickly Please send these exact 6 things for one broken deployment: 1. Prefect Cloud or self-hosted? 2. Worker type (

process

,

docker

,

kubernetes

, etc.) 3. Work pool name 4. Work queue name 5. Last Polled value from the queue page 6. The command used to start the worker Optional but very helpful: 7. one screenshot or copy/paste of the worker logs after restart Then I can tell you the next exact fix, not just general guidance. One important note: I can’t directly fix it for you from here or perform actions in your environment, but I can absolutely help you troubleshoot it live and narrow it down fast if you paste the details above.

That’s a strong clue. If increasing concurrency makes the runs start, then the root cause is very likely: your runs were being blocked by a concurrency limit. So the issue is probably not: - broken flow code - missing logs in the flow itself - scheduler not creating runs It’s more likely one of these limits was too low: - work queue concurrency - deployment concurrency - global concurrency limit And that explains why: - some flows worked - many others became

Late

- no run logs were created Because the runs were waiting to be picked up / submitted, not actually executing. What this means practically A Late run often means: - it was scheduled - but it could not move forward in time If concurrency is full, workers may not start more runs, so scheduled runs wait too long and become

Late

. Why increasing concurrency helped Because you gave the worker/queue more available execution slots, so the scheduled runs could finally move into execution. What to check now You should find which concurrency limit was the bottleneck: 1. Work queue concurrency Check the affected queue: - what was the old limit? - how many runs are typically scheduled at the same time? - how many can your dev infrastructure actually handle? 2. Deployment concurrency If only specific deployments were Late, that deployment may have had a low concurrency cap. 3. Global concurrency limits If multiple deployments share the same limit, they can block each other. Docs: - Global concurrency limits Why there were no logs Because the flow run had not started execution yet. No execution means: - no task logs - no flow logs That’s consistent with a concurrency bottleneck. Recommended fix Don’t just keep increasing concurrency blindly — make sure the new limit matches your dev capacity. Do this: - check average number of scheduled runs at the same time - check how many your worker/infrastructure can really run - set concurrency slightly above normal peak - watch for resource exhaustion on the worker host / cluster If you set it too high, you may fix

Late

runs but overload: - CPU - memory - Docker/Kubernetes capacity - database/API performance So your likely conclusion is Yes — the Late issue in your dev instance is very likely caused by concurrency saturation. Simple interpretation - low concurrency -> runs stay scheduled too long -> become

Late

- higher concurrency -> runs get picked up -> they run normally What I’d do next today For the affected pool/deployments: 1. note the old concurrency value 2. note the new value that works 3. monitor whether runs now go

Running

instead of

Late

4. check worker/infra resource usage 5. identify whether the bottleneck was queue-level or deployment-level If you want, I can help you determine exactly which concurrency setting you should change: - work pool queue concurrency - deployment concurrency - or global concurrency If you paste where you increased it, I can tell you whether that was the correct place.

Yes — if setting the work pool/queue concurrency to unlimited causes all ~200

Late

runs to start, then you’ve confirmed the root cause: the work queue concurrency limit was the bottleneck. And yes, when that limit is removed, Prefect can immediately release the backlog of old scheduled runs, so they all start running. Important: that does not necessarily mean “unlimited” is the right permanent fix — it just proves the queue was throttling execution. First clarification In Prefect 3, concurrency can exist in more than one place, but based on your test, the important one here is almost certainly: - work queue concurrency in the work pool There may also be: - deployment concurrency - global concurrency limits But since changing the work pool/queue concurrency changed behavior immediately, that is your main issue. What is happening This is the likely sequence: 1. many runs get scheduled 2. queue concurrency limit is too low 3. only a small number are allowed to start 4. the remaining scheduled runs wait too long 5. they become

Late

6. when you switch concurrency to unlimited, the queue releases them all That’s why you suddenly see all Late runs start. How to fix it properly You want to do two things: 1. choose a realistic concurrency limit 2. clear the existing Late backlog safely Recommended fix today Option A — safest Instead of

unlimited

, set the queue concurrency to a higher but controlled value. For example, if the old limit was

5

, try: -

20

-

30

-

50

Pick a number your dev infrastructure can actually handle. Why: - unlimited may overload your worker host, Docker daemon, k8s cluster, or database - a controlled increase prevents another massive backlog Option B — if dev environment is small Keep concurrency moderate and reduce how many runs get scheduled at once: - spread schedules apart - reduce frequency - separate heavy flows into another queue/pool What to do with the 200 Late runs Since they were scheduled earlier, you probably do not want all 200 to run at once unless that is intentional. You have two choices: Choice 1: let them run Only do this if: - old runs are still useful - running all of them won’t overload dev Choice 2: cancel stale Late runs This is often better in dev if those old scheduled runs are no longer needed. Then only future runs execute. Best practical approach for today Step 1 Set concurrency to a reasonable finite number, not unlimited. Example: - if unlimited causes chaos, lower it to something sustainable Step 2 Decide whether old Late runs still matter. If not, cancel the stale ones. Step 3 Watch whether new scheduled runs now start on time. If new runs stop becoming Late, you’ve solved the main issue. How to know what number to use Choose based on: - how many workers you have - worker type - available CPU/memory - how heavy the flows are - whether they use Docker/Kubernetes/process execution A rough rule: - if each run is light -> higher concurrency is okay - if each run is heavy -> keep it lower If you ask “how do we permanently fix it?” The permanent fix is usually one of these: - increase queue concurrency to match actual demand - add more workers / capacity - split workloads across multiple queues or pools - reduce schedule frequency - separate heavy and light flows Very likely best architecture If alerts, call processing, and data transformation are all sharing one queue, that can cause backlog. A better setup may be: - one queue for alerts - one queue for call processing - one queue for transformations That way one busy group doesn’t block the others. Why this matters Right now, one queue with a low limit means: - all scheduled runs compete for the same slots - less important work can block more important work - backlog becomes Late What I recommend you do today 1. Do not leave it unlimited unless you’re sure dev can handle it 2. Set a higher finite concurrency

3. Cancel stale Late runs if they are no longer useful 4. Watch new runs for the next few schedule cycles 5. If backlog returns, either: - increase concurrency more - add capacity - split queues by workload Short answer to your question Yes — based on what you observed, the issue is the work queue concurrency limit being too low for the number of scheduled runs. And the fix is: - raise that concurrency to a sustainable number - optionally split the workload across queues/pools - clear old Late runs if they are stale If you want, I can help you decide the right concurrency number and whether you should: - keep one pool and raise limit - create multiple queues - or cancel the 200 stale runs first If you tell me: - worker type - current concurrency value - how many flows typically start at the same time - whether those 200 old runs are still needed I can suggest the safest fix for today.

Yes — Pending flow runs can count toward work queue concurrency. In practice, for work queue/pool capacity, Prefect generally treats active slots as including: -

Pending

-

Running

So it’s not only Running that matters. That means this can happen: 1. a worker picks up runs 2. they move to

Pending

3. they are still occupying queue capacity 4. new scheduled runs cannot be picked up 5. those new runs stay

Scheduled

long enough to become

Late

So if you have many

Pending

runs, they can absolutely contribute to the concurrency bottleneck. Short answer -

Running

counts -

Pending

can also count -

Late

itself is not the state you should think of as consuming execution, but it appears because slots are blocked elsewhere Why this matters in your case Since you said removing the concurrency limit caused many old Late runs to start, it strongly suggests the queue was full of active slots, likely from: -

Running

runs - possibly also

Pending

runs Practical takeaway When debugging queue concurrency, do not only count

Running

. Also look at: -

Pending

- long-stuck

Pending

runs - whether infrastructure is slow to start Because stuck

Pending

runs can consume concurrency and create more

Late

runs behind them. If you want, I can help you with a simple rule for your team: which states to watch in UI when diagnosing concurrency vs Late runs.