What you saw is expected for a process worker, even if it’s frustrating. Short version: with a process work pool, the worker itself is responsible for noticing that the child process exited and then reporting the final state back to Prefect. If the whole machine reboots, the worker disappears before it can report

Crashed

, so the flow run can be left in

Running

. Why this happens: - A process worker starts your flow as a local subprocess - The worker waits for that subprocess to finish - When it finishes, the worker checks the exit code - If the exit code is bad, the worker proposes a

Crashed

state If the machine reboots mid-run: - the flow subprocess dies - the worker dies too - no one is left to send the state update So Prefect Cloud/Server still has the last known state:

Running

. A few important details: - Worker heartbeats only tell Prefect whether the worker is online/offline - An offline worker does not automatically cause its in-flight flow runs to become

Crashed

- For process workers, there isn’t an independent infrastructure monitor that can confirm “this run definitely died” That’s the key difference versus infrastructure-backed execution like Kubernetes/ECS: - those systems have an external job object / platform state - Prefect can often infer failure from the infrastructure state - with a local process worker, the worker host is the infrastructure So in your case: - worker machine rebooted - worker stopped heartbeating and may show offline - flow run stayed

Running

because the crash detection path never completed What to do about it If this failure mode matters, your best options are: 1. Use more durable infrastructure-backed execution - Kubernetes, ECS, etc. - Better when you need the platform to survive worker restarts/reboots 2. Add monitoring for stale

Running

runs - detect runs that have been

Running

too long without logs / progress - mark them failed or investigate manually 3. Run workers on more stable hosts / with restart supervision - helps reduce this class of issue, though it won’t fully solve lost in-flight local processes 4. Design flows to be idempotent / retryable - so if a host dies, rerunning is safe Relevant docs: - Workers concept docs - Work pools and infrastructure concepts If you want, I can also help you with either: - a recommended deployment pattern to avoid this for process workers, or - a small cleanup script/automation that detects stale running flow runs and marks them accordingly.

The supported way is to set the flow run state via the API/Python client. There isn’t currently a built-in CLI command or UI button to manually mark a flow run as failed. Recommended approach

from prefect.client.orchestration import get_client
from prefect.states import Failed

FLOW_RUN_ID = "your-flow-run-id"

with get_client(sync_client=True) as client:
    result = client.set_flow_run_state(
        flow_run_id=FLOW_RUN_ID,
        state=Failed(message="Worker machine rebooted; marking run as failed manually."),
        force=True,
    )
    print(result)

If you prefer async:

from prefect.client.orchestration import get_client
from prefect.states import Failed

FLOW_RUN_ID = "your-flow-run-id"

async def main():
    async with get_client() as client:
        result = await client.set_flow_run_state(
            flow_run_id=FLOW_RUN_ID,
            state=Failed(message="Worker machine rebooted; marking run as failed manually."),
            force=True,
        )
        print(result)

A couple notes: - Use

force=True

so Prefect accepts the state change even though the run is currently

Running

- This is the cleanest way to deal with a stuck run after lost worker/process infrastructure - You’ll need your normal Prefect API connection configured (

PREFECT_API_URL

, and auth if using Cloud) If you want, I can also give you: - a one-off script that finds all stale

Running

runs and marks them failed, or - a raw REST API example with

curl

.