Great question. Here’s how Prefect 3.x handles crashes, retries, and “transaction semantics,” plus links to the right docs and source. High-level behavior - Crashes vs failures - Failures are code-level exceptions. They transition to Failed and, if you’ve configured retries, Prefect automatically schedules an AwaitingRetry for the next attempt. - Crashes are infrastructure-level interruptions (SIGKILL/SIGTERM/OOM/container eviction). Prefect records a Crashed terminal state. Crashed runs do not automatically retry; you resubmit them manually or via Automations. - Heartbeats and zombie detection - Flow runs send heartbeats periodically. If a container dies hard, the run may linger as Running until heartbeats stop; an Automation can then mark it Crashed and take a follow-up action (e.g., notify, retry, or rerun a deployment). - Transactions and idempotency - Prefect’s Transaction API lets you wrap side effects with an idempotent key and commit/rollback semantics. If a crash occurs before commit, nothing is written. On retry, the transaction can detect a committed result for the same key and skip re-execution. - Result persistence - With

persist_result=True

, Completed results are written to result storage. On retry or re-run, Prefect can reuse them (or your Transaction can) to avoid redoing work. What happens if the container crashes mid-flow? - If the runner can observe a non-zero process exit, it proposes a Crashed state with an explanation mapped from the exit code (e.g., OOM). - If the process dies without reporting state, heartbeats will stop; an Automation can mark the run Crashed (a common “zombie flow” safeguard). - Retrying after a crash: - Automatic: not triggered for Crashed. You need either an Automation that schedules a rerun or a manual retry. - Manual: use `prefect flow-run retry <id_or_name>`; if the run is tied to a deployment, it’ll be re-scheduled for a worker to pick up. - On retry, in-memory progress is gone. Any work not persisted/committed will re-run. Use Transactions and/or

persist_result=True

to make re-execution safe and idempotent. Docs - States overview: States (Crashed vs Failed) - Retries (flows & tasks): How to configure retries - Detecting zombie flows (missing heartbeats): Detect zombie flows with Automations - Transactions (idempotent side effects): Transactions - Results and persistence: Results and persistence Relevant source code (GitHub) - State definitions and helpers (includes

Crashed

,

AwaitingRetry

): prefect/states.py - Flow engine (heartbeats and flow execution lifecycle): prefect/flow_engine.py - Runner observing process exit and proposing Crashed: prefect/runner/runner.py - Infrastructure exit code mapping (OOM/SIGKILL explanations): _infrastructure_exit_codes.py - Orchestration policy (retry rules for Failed → AwaitingRetry): server/orchestration/core_policy.py - Flow heartbeat settings: settings/models/flows.py - Transactions API: prefect/transactions.py - Result store and persistence control: prefect/results.py

- Worker/foreman heartbeat monitoring: server/services/foreman.py Manual retry CLI (verified)

prefect flow-run retry --help

Key points: - Works for terminal runs; if there’s a deployment, Prefect schedules the retry and a worker will pick it up. - If there’s no deployment, provide

--entrypoint

and it will run locally. Example: making a re-run safe with Transactions and result persistence

from prefect import flow, task, Transaction

@task(retries=3, retry_delay_seconds=10, persist_result=True)
def do_side_effectful_work(x: int) -> int:
    # The transaction guards the side effect with an idempotent key
    with Transaction(key=f"do_side_effect-{x}"):
        # e.g., write to a table, upload a file, call an API, etc.
        # Only committed on success; on crash/exception, no commit.
        return x * 2

@flow(persist_result=True)
def my_flow():
    a = do_side_effectful_work(1)
    b = do_side_effectful_work(2)
    return a + b

- If the container crashes after

a

commits but before

b

, a retry will detect the committed transaction for

a

and skip redoing it;

b

will run. Would you like a template Automation that: - marks stale Running flow runs as Crashed when heartbeats stop, and - automatically triggers a deployment run to replace them?