Thanks for the detailed context — that helps a lot. Short answer This behavior matches a known scheduler bug in early 3.6.x where runs triggered by multiple schedules (or many deployments created at once) could be marked Late and then never transition to Pending/Running. It was fixed in 3.6.19 (“Fix scheduler to check per-schedule run counts for multi-schedule deployments”). Upgrading to ≥ 3.6.19 should resolve it. - Release notes callout: “Fix scheduler to check per-schedule run counts for multi-schedule deployments” https://github.com/PrefectHQ/prefect/pull/20699 - 3.6.19 release notes: https://github.com/PrefectHQ/prefect/releases (see “Fix scheduler to check per-schedule run counts…”) - Full changelog 3.6.18 → 3.6.19: https://github.com/PrefectHQ/prefect/compare/3.6.18...3.6.19 Why this fits your symptoms - It occurs when many schedules fire at once or you bulk-create dynamic deployments. - No worker-pool/queue concurrency limits doesn’t help; the bug lives in the scheduler’s per‑schedule run accounting, so Late runs can get “stuck” and never be picked up. What to do 1) Tell me your exact patch version (prefect 3.6.x). If it’s < 3.6.19, please upgrade to the latest 3.6.x. 2) After upgrading, cancel or reschedule any stuck Late runs (or let them run if appropriate). 3) Double-check the late-runs service is healthy:

curl -s http://<your-server-host>:4200/api/admin/services | jq '.[] | select(.name=="late_runs")'

Optional quick workaround (if you need to unblock immediately) - Manually transition stuck Late runs to Pending; workers should pick them up right away:

from prefect.client.orchestration import get_client
from prefect.server.schemas import filters, states as server_states

async def bump_late_to_pending():
    async with get_client() as client:
        late = await client.read_flow_runs(
            flow_runs_filter=filters.FlowRunFilter(
                state=filters.FlowRunFilterState(name={"any_": ["Late"]})
            ),
            limit=1000,
        )
        for run in late:
            try:
                await client.set_flow_run_state(run.id, server_states.Pending())
                print(f"Moved {run.id} to Pending")
            except Exception as e:
                print(f"Skip {run.id}: {e}")

import asyncio; asyncio.run(bump_late_to_pending())

A couple of sanity checks (just in case) - Make sure the worker is connected to the correct work pool and is polling (you should see it logging “Checking for flow runs…” regularly). - Confirm the affected deployments are assigned to the same work pool/queue that the worker is serving. If you can share: - Your exact Prefect version (output of

prefect version

) - Whether these deployments have multiple schedules configured I can confirm the match to the 3.6.19 fix and suggest anything else specific to your setup.

@Marvin ^

Got it — this matches a known scheduler bug in early 3.6.x that leaves runs stuck in LATE when many schedules fire at once or when you create deployments in bulk. It was fixed in 3.6.19. What to do - Upgrade to Prefect ≥ 3.6.19 on all components (server and workers). This includes the fix “Fix scheduler to check per-schedule run counts for multi-schedule deployments” https://github.com/PrefectHQ/prefect/pull/20699. See 3.6.19 notes: https://github.com/PrefectHQ/prefect/compare/3.6.18...3.6.19 - If you’re on Prefect Server (self-hosted): update the server image/tag and your workers to the same patch. - If you’re on Prefect Cloud (managed API): just upgrade your workers/CLI. - Sanity checks after upgrade: - Verify the late-runs service is healthy:

curl -s http://<your-server-host>:4200/api/admin/services | jq '.[] | select(.name=="late_runs")'

- Confirm your worker is polling the correct work pool/queue. Quick unblock (optional) - Manually bump stuck Late runs to Pending so workers pick them up:

from prefect.client.orchestration import get_client
  from prefect.server.schemas import filters, states as server_states

  async def bump_late_to_pending():
      async with get_client() as client:
          late = await client.read_flow_runs(
              flow_runs_filter=filters.FlowRunFilter(
                  state=filters.FlowRunFilterState(name={"any_": ["Late"]})
              ),
              limit=1000,
          )
          for run in late:
              try:
                  await client.set_flow_run_state(run.id, server_states.Pending())
                  print(f"Moved {run.id} to Pending")
              except Exception as e:
                  print(f"Skip {run.id}: {e}")

  import asyncio; asyncio.run(bump_late_to_pending())

Could you share: - Your exact Prefect version (output of

prefect version

) - Whether the affected deployments use multiple schedules If you’re on < 3.6.19 and using multiple schedules (or fired a lot of schedules at once), upgrading should resolve this.

Thanks for pushing back — the “gradually gets worse over days until everything is LATE and stuck” pattern is actually an even stronger signal of a specific 3.6.x bug: Most likely root cause - A concurrency-lease handling bug when no deployment concurrency limits are set. Even with “unlimited,” the orchestration policy was still attempting to acquire/renew/release leases. Over time this created bad/stale lease bookkeeping that blocked SCHEDULED→PENDING transitions — so Late runs accumulated and then stopped progressing. - This was fixed to “skip lease renewal/release when no concurrency limits exist” in 3.6.18. Link: https://github.com/PrefectHQ/prefect/pull/20666 - Separately, a scheduler bug affecting deployments with multiple schedules was fixed in 3.6.19. Under load (lots of triggers at once), it could leave runs in Late that never advanced. Link: https://github.com/PrefectHQ/prefect/pull/20699 (also called out in the 3.6.19 release notes: https://github.com/PrefectHQ/prefect/compare/3.6.18...3.6.19) Why it looks “gradual” - With no explicit concurrency limits, the lease bug doesn’t bite immediately — it degrades as more schedules/runs flow through the system. After a few days, the orchestration rules that should move Late→Pending can’t secure “slots,” so nothing progresses and you see a wall of Late runs. What I recommend 1) Upgrade all components to ≥ 3.6.19 (server and workers). That includes both the lease fix (3.6.18) and the per-schedule accounting fix (3.6.19): - PR 20666: https://github.com/PrefectHQ/prefect/pull/20666 - PR 20699: https://github.com/PrefectHQ/prefect/pull/20699 - 3.6.19 notes: https://github.com/PrefectHQ/prefect/compare/3.6.18...3.6.19 2) Restart the API/server and workers after upgrade. This resets the Docket services (scheduler/late_runs) and clears any stuck in-memory state. 3) Clean up/backlog control (one-time): - Either cancel the oldest Late runs en masse, or bump them to Pending so workers pick them up. Example to bump:

from prefect.client.orchestration import get_client
     from prefect.server.schemas import filters, states as server_states

     async def bump_late_to_pending():
         async with get_client() as client:
             late = await client.read_flow_runs(
                 flow_runs_filter=filters.FlowRunFilter(
                     state=filters.FlowRunFilterState(name={"any_": ["Late"]})
                 ),
                 limit=1000,
             )
             for run in late:
                 try:
                     await client.set_flow_run_state(run.id, server_states.Pending())
                     print(f"Moved {run.id} to Pending")
                 except Exception as e:
                     print(f"Skip {run.id}: {e}")

     import asyncio; asyncio.run(bump_late_to_pending())

- If you have very chatty schedules, consider temporarily pausing schedules, clearing backlog, then resuming. 4) Sanity checks after upgrade: - Late-runs service healthy:

curl -s http://<your-server-host>:4200/api/admin/services | jq '.[] | select(.name=="late_runs")'

- Worker is polling the right work pool/queue and regularly logging “checking for flow runs…” - Optional: grep API logs for “SecureFlowConcurrencySlots” or “lease” warnings; they should quiet down after upgrade. If you can share: - Exact Prefect version on server and workers now (output of

prefect version

) - Whether the affected deployments have multiple schedules on a single deployment I can confirm which fix applies most directly to you. But given your “degrades over days” description and “no concurrency limits anywhere,” upgrading to ≥ 3.6.19 has been the definitive fix for teams seeing this.

Yes — it’s relevant for self‑hosted Server as well. That change lives in the API server’s orchestration layer (the same engine used by both Prefect Cloud and self‑hosted Prefect Server). The bug was that the server would still attempt to acquire/renew/release “concurrency leases” even when no deployment concurrency limits existed; over time this could block SCHEDULED→PENDING transitions and leave runs piling up in LATE. The fix makes the server skip that lease path entirely when there are no limits. - PR: Skip lease renewal/release when no concurrency limits exist (included in 3.6.18) - Related scheduler fix (multi‑schedule deployments under load): Fix scheduler to check per‑schedule run counts (in 3.6.19) What this means for you (self‑hosted) - Upgrade your self‑hosted API server to ≥ 3.6.19 (and upgrade workers too). This ensures you have both the lease fix (3.6.18) and the scheduler fix (3.6.19). - After upgrade, restart the API/server and workers so Docket services (scheduler/late_runs) start fresh. - Optional sanity check:

curl -s http://<your-server-host>:4200/api/admin/services | jq '.[] | select(.name=="late_runs")'

If you can share your exact server and worker versions (

prefect version

), I can confirm whether you’re on the affected range.

Great question. Here’s the “how” at a high level, tied to the orchestration flow in Prefect 3.x: - Late is still a Scheduled state. A run becomes Pending only when a worker asks the server for work and the server assigns that run. - On each assignment attempt, the server runs a concurrency check and (if limits exist) acquires/renews a “lease” before allowing Scheduled/Late → Pending. - The bug fixed in 3.6.18 caused the server to still go through the lease acquire/renew/release path even when there were no deployment concurrency limits. Over time, that path could: - Create/retain stale lease bookkeeping or hit renewal/release errors on “non-existent” limits - Cause the assignment step to abort or to conclude “no capacity” due to bad bookkeeping - Return “no work” to workers, leaving the run in Late (since the only time it changes to Pending is during assignment) Why it gets worse over days - Initially, many runs still get through; the bad lease bookkeeping is small. - As more schedules fire (or you bulk-create deployments), more assignment attempts hit the buggy path. Stale/erroneous lease state accumulates and the server increasingly refuses to assign Late runs. - Eventually, workers keep polling but receive “no work,” Late runs never get a Pending assignment, and the backlog becomes a wall of Late. Two relevant fixes - Skip lease renewal/release when no concurrency limits exist (3.6.18): PR 20666 - Fix scheduler per‑schedule run counts for multi‑schedule deployments (3.6.19): PR 20699 - 3.6.19 release notes (mentions the scheduler fix): 3.6.19 changelog What you’ll observe in logs when this happens - Workers poll regularly but often get “no assignments.” - The same runs stay in Late; scheduled times keep passing, but there’s no Pending transition because the assignment step is being blocked/aborted by the faulty lease path. Practical takeaway - Upgrading your self‑hosted server and workers to ≥ 3.6.19 removes both the unnecessary lease path (when unlimited) and the multi‑schedule accounting bug, so Late runs can be assigned normally again. - After upgrading, restart the API/server and workers so the scheduler and late_runs services restart cleanly. - If you need an immediate mitigation before upgrading, you can either: - Manually bump selected Late runs to Pending (workers will pick them up), or - Temporarily set an explicit, realistic deployment‑level concurrency limit (e.g., 50/100) so the server takes the “bounded‑limit” lease path (which avoids the unlimited‑path bug in older versions). If you want, share your exact server/worker versions (

prefect version

) and whether the affected deployments have multiple schedules — I can confirm which fix applies most to your case.