Short answer: it will end up in a CRASHED state, but how quickly that happens depends on what exactly got OOMKilled. - If only the flow subprocess is killed (common when the subprocess hits the pod’s memory limit), the process worker notices the child exited with SIGKILL and immediately proposes a CRASHED state for the flow run with a “high memory usage” message. - If the entire pod is OOMKilled, Prefect won’t get a chance to update the state. The flow run will remain RUNNING until you detect it as a “zombie” and mark it CRASHED via heartbeats/automation. In Prefect 3.x, flow-run heartbeats are off by default; enable them by setting

PREFECT_RUNNER_HEARTBEAT_FREQUENCY

(>= 30s) and add the recommended automation. That typically marks the run CRASHED ~90s after the last heartbeat. Important: CRASHED runs do not trigger flow retries. Retries only apply to FAILED runs; you’d need an automation to resubmit a new run if you want automatic recovery from crashes. Useful links: - States (FAILED vs CRASHED) - Retries (only on FAILED) - Detect zombie flows with heartbeats and automation If you’d like, I can share a quick snippet for enabling runner heartbeats and the automation to mark and/or resubmit crashed runs.

Great questions. There are two parts here: auto-retrying CRASHED runs (e.g., from OOMKills) and what happens when worker pods are scaled down. Part 1 — Retrying CRASHED runs (OOMKilled) - Prefect treats OOMKills as CRASHED, which is terminal and does not trigger built‑in flow retries. You can’t transition a CRASHED run into AwaitingRetry. - Recommended pattern: use an Automation that listens for the CRASHED event and submits a new run of the same deployment. - Use a RunDeployment action. Scope it to the specific deployment(s) to avoid loops. - Optionally add a short delay and a notification. - If you need idempotency (avoid duplicate resubmits), include an idempotency key or add guards in your flow (e.g., check a run label/parameter). Example (conceptual):

Automation:
  Trigger:
    expect: ["prefect.flow-run.Crashed"]
    match: {"prefect.deployment.id": ["<your-deployment-id>"]}
    posture: Reactive
  Actions:
    - RunDeployment:
        source: "selected"
        deployment_id: "<your-deployment-id>"
        schedule_after: "300s"
    - SendNotification: ...

Also consider enabling flow-run heartbeats and the “detect zombie flows” Automation so pods that die before reporting CRASHED get marked CRASHED quickly: - Set

PREFECT_RUNNER_HEARTBEAT_FREQUENCY

(>= 30s). - Use the Detect Zombie Flows recipe to mark missing-heartbeat runs as CRASHED, then your “resubmit” Automation fires. Links: - States (FAILED vs CRASHED): States - Retries (only on FAILED): Retries - Detect zombie flows: Detect zombie flows Part 2 — If HPA scales worker pods down - Runs are not lost. Prefect’s work-queue + DB row locking uses “select for update skip locked” so only one worker claims a run at a time. If a worker disappears, its DB connection drops and any locks it held are released; the run becomes available for another worker on the next poll. - Concurrency limits are protected by leases with TTL. If a worker dies while holding a concurrency lease, the Repossessor service revokes expired leases and frees capacity; runs waiting in AwaitingConcurrencySlot become eligible again. - Bottom line: other workers will pick up the work; there’s usually a short handoff window (seconds to a few minutes depending on poll intervals and lease TTL). Practical tips - For OOMKills, set a conservative memory limit or add infra retries at your orchestrator (e.g., Kubernetes job/controller retry) if you need infra-level retries before resubmitting through Prefect. - Use a targeted Automation per deployment to resubmit CRASHED runs; avoid a global “resubmit everything” rule to prevent accidental loops. - If you want to minimize downtime on scale-downs, run multiple worker replicas per work pool so another pod is always there to pick up released work. If you want, I can draft the exact Automation JSON/YAML (or Python client code) for your deployment(s) and include a notification step.

Likely causes: - Prefect doesn’t auto-mark “missing” runs as Crashed. Setting

PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30

only makes the runner emit heartbeat events; you still need an Automation to detect missing heartbeats and change the state to Crashed. - If the pod is OOMKilled, the process worker can’t report the final state. Without the “zombie flow” Automation, the run just sits in RUNNING. - The env var must be present in the flow run’s execution environment. If it’s only on the worker Deployment but your runs execute in a different container/pod, no heartbeats will be emitted. What to do: 1) Verify heartbeats are actually being emitted - Open an affected flow run and look at its Events timeline for

prefect.flow-run.heartbeat

events every ~30s. If you don’t see them: - Ensure

PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30

is set in the container that actually executes the flow (for process worker: the worker pod; for Kubernetes worker: the job pod template). - Confirm you’re on Prefect 3.1.8+ (you’re on 3.6.6, so good). 2) Add the “Detect Zombie Flows” Automation - This marks flows as Crashed when heartbeats stop (e.g., OOMKill or node/pod eviction). Here’s the pattern: Docs: Detect zombie flows Example Automation (Proactive trigger on missing heartbeats):

{
  "name": "Mark zombie flows as Crashed",
  "trigger": {
    "type": "event",
    "posture": "proactive",
    "after": [
      "prefect.flow-run.heartbeat"
    ],
    "expect": [
      "prefect.flow-run.heartbeat"
    ],
    "within": "90s",
    "threshold": 1,
    "match": {
      "prefect.deployment.id": [
        "<your-deployment-id>"
      ]
    }
  },
  "actions": [
    {
      "type": "change-flow-run-state",
      "state": "CRASHED",
      "message": "No heartbeat in 90s; marking as crashed (likely OOMKill or infra interruption)."
    }
  ]
}

Optional: add a second Automation to resubmit the run when it’s marked Crashed:

{
  "name": "Resubmit on Crashed",
  "trigger": {
    "type": "event",
    "posture": "reactive",
    "expect": [
      "prefect.flow-run.Crashed"
    ],
    "match": {
      "prefect.deployment.id": [
        "<your-deployment-id>"
      ]
    }
  },
  "actions": [
    {
      "type": "run-deployment",
      "source": "selected",
      "deployment_id": "<your-deployment-id>",
      "schedule_after": "300s"
    }
  ]
}

Quick checklist: - Heartbeats present in Events timeline? - Zombie Automation installed and scoped to the right deployment(s)? - If using Kubernetes worker: is

PREFECT_RUNNER_HEARTBEAT_FREQUENCY

injected into the job pod’s env? Setting it only on the worker pod won’t affect the job container. - For process worker: if the entire worker pod is OOMKilled, immediate crash reporting is impossible; the Automation is what cleans up the RUNNING state. If you share: - Worker type (process vs kubernetes) - Where you set

PREFECT_RUNNER_HEARTBEAT_FREQUENCY

- Whether you see heartbeat events on those runs …I can tailor the exact Automation and, if needed, the k8s env injection for your worker/job pods.