<@ULVA73B9P> We have various flow runs which are s...
# ask-marvin
c
@Marvin We have various flow runs which are stuck in various non terminal states. Some have been like this for multiple days. What does this mean?
m
thought for 140.6 seconds
It means those runs have not reached a final outcome yet — they’re still in states where Prefect thinks they may continue, retry, resume, or be cleaned up. Common examples of non-terminal states: -
Scheduled
/
Late
— supposed to start, but no worker picked them up -
Pending
— a worker picked them up, but infrastructure hasn’t started cleanly -
Running
— marked as active, but may actually be stuck or orphaned -
Paused
— waiting for manual resume or timeout -
AwaitingRetry
— waiting for the next retry time -
AwaitingConcurrencySlot
— blocked by a concurrency limit -
Cancelling
— cancellation started, but cleanup hasn’t finished So if runs have been there for days, that usually means one of these is happening: Most common causes - No healthy worker is polling the work pool → runs stay
Late
- Worker can’t start infrastructure → runs stay
Pending
- Flow process crashed without reporting back → runs stay
Running
- A pause/retry/concurrency condition is blocking progress - Cancellation got stuck during infrastructure cleanup What to check first 1. Look at the exact state name for one stuck run in the UI 2. Open the run logs — often the fastest clue 3. Check your workers and work pool health 4. Check whether the run is blocked by retries, pauses, or concurrency limits How to interpret the state -
Late
: worker/pool problem -
Pending
: infrastructure submission problem -
Running
for days with no new logs
: likely orphaned/zombie -
Paused
: expected until resumed -
AwaitingRetry
: expected until retry time -
AwaitingConcurrencySlot
: expected if limits are full -
Cancelling
: cleanup likely failing or hung Recommended docs - States - Workers - Worker healthchecks - Detect zombie flows - Global concurrency limits - Tag-based concurrency limits If you want, send me: - the exact state names you’re seeing - whether you’re using Prefect Cloud or server - what infrastructure you run on (process, docker, kubernetes, ecs, etc.) and I can help narrow down what each stuck state likely means in your setup.
c
@Marvin can I manually set zombie flows as crashed?
m
thought for 9.4 seconds
Yes — you can manually mark a zombie/orphaned flow run as
Crashed
. That’s a reasonable cleanup step when: - the underlying process/container/pod is definitely gone - the run is still stuck in
Running
or another non-terminal state - you don’t expect it to recover on its own A zombie flow usually means Prefect still thinks the run is active, but the actual execution environment died or disappeared. Best practice before doing it 1. Confirm the infrastructure is really gone - container exited - pod was evicted/deleted - process is no longer running 2. Check the run logs to make sure it’s not just slow 3. Then mark it
Crashed
If you need this regularly, Prefect has guidance for automating zombie detection: Detect zombie flows If you want, I can also show you the API/Python way to manually set a flow run to
Crashed
, since that’s often the most reliable approach.
c
@Marvin how do I manually set a zombie flow as crashed in Prefect Cloud UI?
m
thought for 7.2 seconds
Prefect Cloud UI does not currently provide a direct “mark as `Crashed`” button for a flow run. In the UI, you can typically cancel runs, but if you specifically want the state to be
Crashed
, you’ll need to do it via the API/Python client instead. A common pattern is:
Copy code
from uuid import UUID
from prefect import get_client
from prefect.states import Crashed

flow_run_id = UUID("YOUR-FLOW-RUN-ID")

async def main():
    async with get_client() as client:
        await client.set_flow_run_state(
            flow_run_id=flow_run_id,
            state=Crashed(message="Manually marked as crashed after becoming a zombie.")
        )

import asyncio
asyncio.run(main())
That said, before doing this, make sure the underlying execution environment is actually gone; otherwise you can create a mismatch where the flow is still doing work but Prefect thinks it crashed. Related docs: - Detect zombie flows - States If you want, I can give you: - a one-off script you can run locally against Prefect Cloud - a snippet to do this for multiple stuck flow runs - or an automation so zombie runs get marked
Crashed
automatically