Great question. Here’s how Prefect 3.x handles run states around retries and restarts, and how long runs can stay in each state. What you’re seeing when you “restart” a crashed run - When you click Retry on a crashed flow run (or use the CLI), Prefect transitions the same run out of its terminal state and schedules it again. - If your flow has retries configured (e.g.,

@flow(retries=..., retry_delay_seconds=...)

), Prefect will typically put the run into an AwaitingRetry state with a scheduled_time set in the future (now + retry_delay). That’s why you may see “AwaitingRetry for quite a while.” - After the scheduled_time arrives, a worker picks it up and the run moves to Running (specifically “Retrying” for clarity). Typical state flow when you retry a crashed run - Crashed (terminal) - Retry action (UI/CLI) → Scheduled or AwaitingRetry - At scheduled_time → Pending (worker submission/provisioning) - Running (may show as Retrying) - Completed or Failed (or Crashed again) How long a run can stay in each relevant state - AwaitingRetry - Duration: exactly until its scheduled_time, which is computed from your retry delay (plus any jitter/backoff you configured). - If not otherwise configured, flow-level retries default to immediate retry (0 seconds). If you set a delay like 300 seconds, it’ll wait ~5 minutes here before trying again. - It will remain here until scheduled_time arrives; after that, it behaves like a normal scheduled run. - Scheduled and Late - Scheduled: until its scheduled_time arrives and a worker picks it up. - Late: if a run is still Scheduled 15 seconds after its scheduled_time, it’s marked Late. Late is just a label; the run will still start as soon as a worker picks it up. - There is no hard maximum; a run can remain Scheduled/Late indefinitely if no workers are available or the queue is paused. (Late threshold defaults to ~15s on the server.) - Pending - Duration: until infrastructure is provisioned and the process starts. No hard cap; can sit here if provisioning is slow or blocked. - Running / Retrying - Duration: until completion, failure, crash, or timeout (if you set a timeout). No hard cap otherwise. - Paused / Suspended - Duration: indefinitely unless you set a pause timeout. When the timeout expires, the run will fail automatically. Notes on why “AwaitingRetry for a while” can happen - Flow retry policy: If you’ve set

retry_delay_seconds

at the flow level, each retry is scheduled in the future by that amount. That’s the most common cause. - Concurrency limits: If you’re also using deployment or tag-based concurrency limits, the run can be Scheduled (or AwaitingConcurrencySlot) until a slot opens up, then proceed. - Worker availability: If no worker is available, the run remains Scheduled/Late after its scheduled_time until a worker picks it up. How to inspect what it’s waiting on - From the CLI:

prefect flow-run inspect <FLOW_RUN_ID>

This shows the current state, the state message, and the scheduled_time (for AwaitingRetry/Scheduled), which will tell you exactly when it’ll try to run again. - To manually re-run a terminal run:

prefect flow-run retry <FLOW_RUN_ID>

If the run has a deployment, it’ll be scheduled for a worker to pick up; without a deployment, you can provide

--entrypoint

to run it locally. If you want to reduce the wait - Lower or remove the flow’s

retry_delay_seconds

(e.g.,

@flow(retries=3, retry_delay_seconds=0)

). - If you’re okay with manual reruns being immediate regardless of the retry policy, you can temporarily set

retries=0

at the flow level during your testing. - Check your work pool/queue is not paused and that a worker is polling the queue. If you share a run ID or a screenshot of the State details, I can help pinpoint exactly which setting (retry delay, concurrency, or worker pickup) is driving the delay.