@Marvin What are the various reasons for a flow to be ran late from it's scheduled time?

✅ thought for 1177.7 seconds

most commonly • worker (listening to the right work pool) is not alive • concurrency limits

Great question. In Prefect 3.x, a scheduled flow run can start late for a handful of common reasons. Here’s a concise checklist to help you diagnose: Worker / queue / pool related - No worker is running on the target work pool, or the worker is polling the wrong work queue(s) - Work pool or work queue is paused - Backlog in the queue (more scheduled runs than available worker capacity) - Concurrency limits: - Work pool concurrency limit reached - Work queue concurrency limit reached - Global or tag-based task/flow concurrency limits reached (runs wait in “AwaitingConcurrencySlot”) - Worker cannot claim the run due to mismatched job variables or incompatible base job template Infrastructure startup and provisioning - Slow spin-up of infrastructure (e.g., Kubernetes/ECS jobs pending, Docker image pulls, node provisioning/scale-up) - Long environment bootstrap (large image, dependency installation at runtime, network pulls) - Storage/source checkout delays (e.g., pulling from Git via

flow.from_source(...)

) Scheduling and orchestration - Timezone/DST mismatch in the deployment schedule vs. expectations - Clock skew between your scheduler, worker machine(s), and UI expectations - API/scheduler delays or temporary service connectivity issues - Run is intentionally deferred: - Awaiting concurrency slot - Awaiting retry (backoff) - Paused/Suspended until manually or programmatically resumed Misconfiguration or errors - Work pool misconfiguration (bad job template, missing permissions) preventing start - Work queue filters don’t match your deployment - Run input or automation gating (e.g., awaiting manual input) holding the run About the “Late” state in the UI - “Late” indicates the scheduled start time has passed and the run has not progressed to pending/running yet; it’s a non-terminal scheduled state. Once a worker claims it or a block is cleared, it should move forward. - If a worker claims the run before the scheduled time but the container takes a long time to spin up, you may see “Pending/Running” rather than “Late,” even though the actual work starts later than you expect. Quick ways to diagnose - Verify the worker is running on the correct pool/queues and has capacity:

prefect work-pool inspect <pool-name>
  prefect work-queue ls
  prefect work-queue inspect <queue-id>
  prefect worker start -p <pool-name>  # run a worker if you don’t have one

- Check deployment schedules and timezones:

prefect deployment inspect <flow-name>/<deployment-name>
  prefect deployment schedule ls <flow-name>/<deployment-name>

- Look for concurrency bottlenecks:

prefect work-pool inspect <pool-name>  # pool concurrency
  prefect work-queue inspect <queue-id>  # queue concurrency
  prefect concurrency-limit ls           # tag-based limits
  prefect global-concurrency-limit ls    # global limits

- Peek at runs waiting in a queue:

prefect work-queue read-runs <queue-id>

- Confirm the state timeline and logs in the UI for clear signals like AwaitingConcurrencySlot, AwaitingRetry, Paused, etc. References - States in Prefect 3 - Work pools and workers - Schedules (API reference) If you can share a specific flow/deployment name, work pool type, and what you see in the run’s state timeline, I can help pinpoint the exact cause.