Thanks for the detailed timeline — here’s what’s going on and how to unblock it. High-level mechanics (Prefect 3.x) - “Late” is just a Scheduled variant (state_type=SCHEDULED, name="Late"). It does not consume a concurrency slot. - When a deployment’s concurrency is full and a run is claimed, orchestration rejects PENDING and sets the run to “AwaitingConcurrencySlot” (also SCHEDULED) with a short future scheduled_time; workers will retry later. - After you raise/remove the limit, workers should be able to move those SCHEDULED runs to PENDING on the next poll. Why you’re seeing AwaitingConcurrencySlot → Pending → stuck - Workers only poll SCHEDULED runs. Once a run enters PENDING, a worker “owns” it to submit infrastructure. If the worker crashes/exits between moving the run to PENDING and submitting, the run can sit in PENDING. - There are known edge cases where runs waiting on concurrency are not picked up promptly after the slot opens or after job variables change, leading to a loop between SCHEDULED/AwaitingConcurrencySlot and PENDING (see: - Runs not picked up from AwaitingConcurrencySlot: #16186 - “Server returned a non‑pending state ‘SCHEDULED’” when job variables change: #17452 - Related slow transitions from Late to Pending: #18877) Common causes of “stuck in Pending” - No healthy worker polling the target work pool/queue (or the pool is paused, or has its own concurrency limit). - The worker moved the run to PENDING but failed/aborted before infra submission. - Work-queue mismatch (labels/filters) preventing a worker from ever seeing new SCHEDULED runs — the PENDING ones won’t be retried by other workers. - Edge case around AwaitingConcurrencySlot rechecks after changing limits (issue links above). How to unblock impacted runs now - Check your worker and work pool: - Ensure at least one worker is running/healthy on the pool the deployment targets, and the pool is not paused. - Ensure the work pool itself does not have a concurrency limit set (or raise it). - For runs stuck in PENDING: - Easiest: Cancel the stuck runs, then re-trigger. You can do this in the UI or via CLI:

prefect flow-run cancel <flow-run-id>

- Or, nudge them back to SCHEDULED via API so workers can pick them up again:

from prefect.client.orchestration import get_client
    from prefect.client.schemas import StateCreate, StateType
    import anyio

    async def reschedule(run_id):
        async with get_client() as client:
            await client.set_flow_run_state(
                run_id,
                state=StateCreate(type=StateType.SCHEDULED)
            )

    anyio.run(reschedule, "<flow-run-id>")

- After changing a deployment’s concurrency limit, give workers a short window to re-poll. If you’re unblocking a backlog, a practical pattern is: - Temporarily raise the limit to a large number (e.g., 10000), wait ~30–60s for AwaitingConcurrencySlot runs to re-evaluate, then adjust to your desired setting. Prevention tips - Prefer increasing the limit rather than toggling to “unlimited” when many runs are already in AwaitingConcurrencySlot; then tune it down after the queue clears. - Keep workers running/healthy and watch for errors like “Aborted submission: Server returned a non‑pending state 'SCHEDULED'” (see #17452). - Make sure there’s no separate work pool concurrency limit bottlenecking you. Can you share a bit more so I can pinpoint the root cause? - Prefect version (Cloud or self-hosted) and worker type (process, docker, k8s, ecs, push/pull). - Whether the work pool has a concurrency limit or is paused. - A sample flow-run ID stuck in PENDING and a snippet from the worker logs around the time it was claimed.

- Whether job variables changed between attempts (linked to #17452). With those details, I can correlate against the known issues and suggest the safest fix for your setup.

@Marvin I received the "Aborted submission" error. Let see if I understand it correctly: The issue happens when a Run turn to Pending (picked up by a worker and prepare infra to start the Run). While in Pending, worker is preparing infra, the worker goes offline for some reason. That run will stay in Pending forever

You’re very close, but there are two distinct scenarios here: 1) “Aborted submission: Server returned a non‑pending state 'SCHEDULED'” - This happens when the worker tries to move the run to PENDING but the API rejects it (e.g., deployment/work-pool concurrency is full). The API returns a SCHEDULED variant like “AwaitingConcurrencySlot”, so the worker aborts. This is not a worker-offline problem; it’s a concurrency/eligibility rejection. - Related issues: #16186, #17452. 2) Runs stuck in PENDING forever - Yes, this can happen if a worker successfully moved a run to PENDING and then goes away (crashes, is stopped, or cannot submit). Workers poll SCHEDULED runs; they do not “adopt” PENDING ones. Without another state change, that run can sit in PENDING indefinitely. How to unblock stuck PENDING runs - Cancel and re-run, or set them back to SCHEDULED so a worker can pick them up again:

from prefect.client.orchestration import get_client
from prefect.client.schemas import StateCreate, StateType
import anyio

async def reschedule(run_id):
    async with get_client() as client:
        await client.set_flow_run_state(
            run_id,
            state=StateCreate(type=StateType.SCHEDULED)
        )

anyio.run(reschedule, "<flow-run-id>")

Prevention and triage - Ensure a healthy worker is running on the target work pool/queue (and the pool is not paused, and doesn’t have its own concurrency limit). - If you changed the deployment concurrency limit from a small value to unlimited, briefly bump it to a large number (e.g., 10000) to clear the backlog, wait ~30–60s, then set your desired value. - Consider an Automation: “Flow run stays in Pending for > N minutes” → “Set state: Scheduled” or “Cancel.” This recovers automatically from orphaned PENDING runs. - If you see the “Aborted submission” message, check for: - Deployment/work-pool concurrency rejections - Work pool filters/labels mismatches - Recent job variable changes (linked to #17452) If you can share: - Worker type (process/docker/k8s/ecs; push or pull), Cloud vs self-hosted - Whether the work pool has a concurrency limit or is paused - A sample flow run ID and the worker logs around the time it went PENDING …I can pinpoint which of the two paths you’ve hit and recommend the safest fix for your setup.