In Prefect 3.x, LATE means “this run was scheduled to start, but the scheduled time has passed and it hasn’t actually started yet.” Quick details: - What it is: A Scheduled state (not Running and not Failed). Think of it as “overdue.” - When it happens: The run’s

start_time

has passed but no worker has picked it up yet. This can be brief (e.g., worker poll/startup delay) or persistent. - How it resolves: As soon as a worker picks it up, it will move to Pending/Running as normal. - Impact on retries/cancellation: None by default. LATE doesn’t trigger retries and it isn’t auto-cancelled. In Prefect Cloud you can create an Automation on “Flow run late” to notify or cancel after some time. - Common causes: - No workers online or the work pool is paused - Worker can’t match the work pool/queue or labels - Concurrency limits holding runs - Slow infrastructure start-up or backlog (e.g., cold starts on Vertex AI) - Very tight schedules vs worker poll intervals Docs: States in Prefect If you’re seeing lots of LATE runs in your Vertex AI work pool, I can help you check: - Work pool status (not paused) and concurrency limits - Worker logs for claim attempts - Poll interval and startup time of your Vertex AI base image - That the deployment targets the correct work pool and queue

Great question — these two states mean different things in Prefect 3.x and they can coexist when you launch a large burst of runs. - AwaitingConcurrencySlot: The run has been picked up to start but is blocked by a concurrency policy (e.g., a global concurrency limit). It’s queued by Prefect and will start when a slot opens. - Late: The run’s scheduled start time has passed but it hasn’t been picked up by a worker yet. It will remain Scheduled/Late until a worker claims it. Why you’re seeing both with 1000 runs and a limit of 50: - If you set a work pool “max concurrency” of 50, workers will only run up to 50 at a time. Extra runs typically remain Scheduled and quickly become Late once their scheduled time is in the past because they haven’t been claimed yet. - If you also have a global concurrency policy (or a flow/task-level concurrency constraint) of 50, some runs that get claimed will be put into AwaitingConcurrencySlot until the policy frees a slot. - Net effect: some runs are Late (not yet claimed by any worker), while others are AwaitingConcurrencySlot (claimed but waiting on a concurrency slot). The 50 “Running” are the active ones. How to influence what you see: - Want more runs in AwaitingConcurrencySlot and fewer in Late? Ensure workers are available to claim runs promptly (scale workers or reduce their poll intervals) and use a global concurrency policy to queue starts at the server. The pool limit still caps actual execution at 50, but claimed runs that hit the policy will show AwaitingConcurrencySlot instead of sitting Late. - Want fewer Late runs overall? Stagger or rate-limit scheduling so not all 1000 are due at the same instant, or increase worker capacity to claim runs faster (the pool/global limit will still cap actual concurrency). Docs for reference: - States in 3.x: States - Work pools: Work pools - Global concurrency limits: Global concurrency limits Quick clarification to tailor guidance: is your “concurrency limit of 50” a work pool max concurrency, a global concurrency policy, or both? Since you’re on Vertex AI work pools, I can suggest specific tweaks (poll interval, worker scaling, etc.) once I know which limits you’ve set.