Craig Rose
05/15/2026, 12:01 AMMarvin
05/15/2026, 12:01 AMMarvin
05/15/2026, 12:03 AMScheduled / Late — supposed to start, but no worker picked them up
- Pending — a worker picked them up, but infrastructure hasn’t started cleanly
- Running — marked as active, but may actually be stuck or orphaned
- Paused — waiting for manual resume or timeout
- AwaitingRetry — waiting for the next retry time
- AwaitingConcurrencySlot — blocked by a concurrency limit
- Cancelling — cancellation started, but cleanup hasn’t finished
So if runs have been there for days, that usually means one of these is happening:
Most common causes
- No healthy worker is polling the work pool → runs stay Late
- Worker can’t start infrastructure → runs stay Pending
- Flow process crashed without reporting back → runs stay Running
- A pause/retry/concurrency condition is blocking progress
- Cancellation got stuck during infrastructure cleanup
What to check first
1. Look at the exact state name for one stuck run in the UI
2. Open the run logs — often the fastest clue
3. Check your workers and work pool health
4. Check whether the run is blocked by retries, pauses, or concurrency limits
How to interpret the state
- Late: worker/pool problem
- Pending: infrastructure submission problem
- Running for days with no new logs: likely orphaned/zombie
- Paused: expected until resumed
- AwaitingRetry: expected until retry time
- AwaitingConcurrencySlot: expected if limits are full
- Cancelling: cleanup likely failing or hung
Recommended docs
- States
- Workers
- Worker healthchecks
- Detect zombie flows
- Global concurrency limits
- Tag-based concurrency limits
If you want, send me:
- the exact state names you’re seeing
- whether you’re using Prefect Cloud or server
- what infrastructure you run on (process, docker, kubernetes, ecs, etc.)
and I can help narrow down what each stuck state likely means in your setup.Craig Rose
05/15/2026, 12:44 AMMarvin
05/15/2026, 12:44 AMMarvin
05/15/2026, 12:44 AMCrashed.
That’s a reasonable cleanup step when:
- the underlying process/container/pod is definitely gone
- the run is still stuck in Running or another non-terminal state
- you don’t expect it to recover on its own
A zombie flow usually means Prefect still thinks the run is active, but the actual execution environment died or disappeared.
Best practice before doing it
1. Confirm the infrastructure is really gone
- container exited
- pod was evicted/deleted
- process is no longer running
2. Check the run logs to make sure it’s not just slow
3. Then mark it Crashed
If you need this regularly, Prefect has guidance for automating zombie detection:
Detect zombie flows
If you want, I can also show you the API/Python way to manually set a flow run to Crashed, since that’s often the most reliable approach.Craig Rose
05/15/2026, 12:46 AMMarvin
05/15/2026, 12:46 AMMarvin
05/15/2026, 12:47 AMCrashed, you’ll need to do it via the API/Python client instead.
A common pattern is:
from uuid import UUID
from prefect import get_client
from prefect.states import Crashed
flow_run_id = UUID("YOUR-FLOW-RUN-ID")
async def main():
async with get_client() as client:
await client.set_flow_run_state(
flow_run_id=flow_run_id,
state=Crashed(message="Manually marked as crashed after becoming a zombie.")
)
import asyncio
asyncio.run(main())
That said, before doing this, make sure the underlying execution environment is actually gone; otherwise you can create a mismatch where the flow is still doing work but Prefect thinks it crashed.
Related docs:
- Detect zombie flows
- States
If you want, I can give you:
- a one-off script you can run locally against Prefect Cloud
- a snippet to do this for multiple stuck flow runs
- or an automation so zombie runs get marked Crashed automatically