In Prefect 3.x, you generally don’t “fail” those task runs when the flow’s infrastructure dies — they should be marked

CRASHED

, not

FAILED

. Why:

FAILED

means user code ran and raised an error.

CRASHED

means the run stopped because the execution environment disappeared or became unhealthy (OOM, pod eviction, infra yanked, worker/process crash, etc.). What happens: - If the flow process exits unexpectedly, the flow run is typically marked

CRASHED

. - Any non-terminal child task runs should also be transitioned to

CRASHED

rather than left running/pending. - If the infrastructure disappears silently, Prefect relies on heartbeats / unresponsive-run detection to detect that and mark the flow crashed. So the short answer is: You should not try to mark them as `FAILED`; the correct state is

CRASHED

. A few practical notes: - OOMKilled / pod-level infra failure → treat as

CRASHED

- Worker or pod yanked → treat as

CRASHED

- Actual exception in task code →

FAILED

If your concern is that tasks are being left in

RUNNING

or

PENDING

after the parent flow dies, that’s the thing to fix. In that case: 1. Make sure your flow run crash/unresponsive detection is configured 2. Ensure child task runs are being propagated to a terminal crashed state For unresponsive infra, Prefect supports heartbeat-based detection: - Prefect Cloud has managed unresponsive-run detection - In self-hosted setups, you may need an automation to detect missing heartbeats and mark the flow run

CRASHED

Relevant docs/resources: - Prefect docs - Events and automations - States If you want, I can help with either: 1. how to configure unresponsive/heartbeat detection, or 2. how to make sure child task runs get marked

CRASHED

when a flow pod dies. If you’re seeing tasks stuck in a specific state, share: - Prefect Cloud or self-hosted - your execution environment (Kubernetes worker? ECS? process?) - the state the tasks are stuck in (

RUNNING

,

PENDING

, etc.)

Got it — you’re talking about in-flight child flows/tasks that were executing on a node that disappeared underneath them, so they never get a chance to report their own final state. In that situation, the right handling is still: - the affected flow runs / task runs should end up in

CRASHED

- not

FAILED

- and not left indefinitely in

RUNNING

Why? Because nothing in user code necessarily failed — the runtime vanished. What should happen conceptually If a worker node is torn down: 1. The top-level flow running there stops heartbeating / exits 2. Prefect marks that flow run as

CRASHED

3. Any child task runs or subflow runs that were still non-terminal should also be moved to a terminal state, typically

CRASHED

So the desired model is propagation of the infrastructure crash downward to in-flight children. Important distinction - Subflow run = its own flow run, and should itself become

CRASHED

- Task run = should become

CRASHED

if it was active and its parent flow disappeared - Already terminal children (

Completed

,

Failed

,

Cancelled

, etc.) should remain unchanged If they don’t get updated automatically That usually means you’ve hit the classic “orphaned run” problem: the process died before it could write a final state, and there isn’t yet a cleanup path catching every child run promptly enough. In practice, this is usually handled by one of these mechanisms: 1. Heartbeat/unresponsive-run detection The missing heartbeat marks the run as crashed after a timeout window. 2. Crash propagation / orphan cleanup When the parent flow is recognized as crashed, any still-open child task runs should be moved to

CRASHED

. 3. Operational cleanup automation If some runs still remain stuck as

RUNNING

, create an automation or periodic cleanup process that finds task runs whose parent flow run is

CRASHED

and forcibly transitions them to

CRASHED

. So the practical answer to “how should that be handled?” is: Treat them as orphaned runs caused by infrastructure loss, and reconcile them to

CRASHED

once the parent crash is known. Not

FAILED

, because there’s no evidence the task code itself threw an exception. What I would recommend operationally - Ensure unresponsive run detection is enabled/configured - Verify whether the problematic children are task runs or subflow runs - If they remain stuck, add a cleanup job that: - finds non-terminal task runs - checks whether their parent flow run is terminal/`CRASHED` - force-sets those task runs to

CRASHED

If you want, I can help sketch that cleanup logic. One more nuance: if these are separately deployed subflows running on their own infrastructure, then each subflow run may need its own heartbeat/crash detection too. A parent flow crashing does not necessarily mean an independently running subflow pod also crashed. Docs: - States - Events and automations - Prefect docs If you want, I can give you a concrete pattern for: - Kubernetes workers/pods disappearing - child task runs stuck in RUNNING - or subflows launched from a parent flow

It means Prefect does not automatically infer “this flow pod disappeared, so mark it crashed” unless you have the detection mechanism in place. In practice, that detection is based on flow heartbeats plus an automation that notices when heartbeats stop. Where to find it Prefect Cloud - Open your workspace in Prefect Cloud - Go to Automations - Create an automation that watches for missing

prefect.flow-run.heartbeat

events Self-hosted Prefect server - Open your Prefect UI - Go to Automations - Create the same kind of automation there So this is not usually a “single toggle” under flow settings — it’s configured as an automation. What to configure You want an automation that: - starts watching after a flow heartbeat is seen - expects more heartbeat events - triggers if they stop arriving within some time window - then changes the flow run state, typically to

CRASHED

or

CANCELLED

There’s also a related runtime setting: -

PREFECT_FLOWS_HEARTBEAT_FREQUENCY

- controls how often flow heartbeats are emitted - default is typically

180

seconds So the two pieces are: 1. Heartbeat emission from the running flow 2. Automation that reacts when those heartbeats stop Important caveat Even with unresponsive-run detection, this primarily marks the flow run. If your real goal is “also mark child task runs terminal when the flow pod dies,” that depends on the child-task propagation behavior in your Prefect version/setup. Docs: - Events and automations - States If you want, I can give you: - a step-by-step UI recipe for the automation, or - a Python/API example to create the automation.

That means: Prefect needs some mechanism to decide “this run stopped responding, so mark it

CRASHED

.” That mechanism is usually based on flow heartbeats plus an automation that reacts when heartbeats stop. Where this is configured depends on your setup: ### If you're using Prefect Cloud Look in the UI under Automations. That’s where unresponsive-run / heartbeat-based crash detection is managed. Start here: - Prefect Cloud UI - Events and automations ### If you're using self-hosted Prefect server There usually is not a magic single “enable crash detection” toggle in the UI. Instead, you configure: 1. Heartbeat emission frequency on the flow runner side 2. An automation that detects missing heartbeat events and marks the run crashed The main runtime setting is: -

PREFECT_FLOWS_HEARTBEAT_FREQUENCY

- default:

180

seconds - minimum:

30

seconds Example:

bash
export PREFECT_FLOWS_HEARTBEAT_FREQUENCY=60

That only controls how often heartbeats are sent — not what happens when they stop. ### What the automation does The automation watches for: - a

prefect.flow-run.heartbeat

- then expects another flow event within some window - if nothing arrives, it treats the flow as unresponsive/crashed So when I said “make sure crash/unresponsive detection is configured,” I meant: - Cloud: verify the relevant automation exists/is enabled - Self-hosted: create an automation for missing flow heartbeats ### Important nuance This is for flow runs. If your concern is task runs created inside that flow getting stranded after the node disappears, heartbeat detection marks the flow run crashed first. Then you need child-run reconciliation so those in-flight task runs don’t remain stuck. ### What to check next Can you confirm whether you are on: - Prefect Cloud, or - self-hosted/server? If you tell me that, I can give you the exact place in the UI or the exact automation/config pattern to use.

Thanks — that clarifies it. You’re not asking about detecting the top-level flow crash. You’re saying: - top-level deployment flow run is correctly marked terminal - nested subflow run and its task runs that were executing in the same process/pod are left orphaned in non-terminal states - and you want to know how that should be handled Short answer: those child runs should be marked

CRASHED

, and if they are not, that’s a gap/bug in run reconciliation rather than something you configure with heartbeats. Because: - the parent flow’s infra died - the subflow and tasks did not fail by raising Python exceptions - they also did not complete/cancel cleanly - so

CRASHED

is the correct terminal state What you are describing is not solved by configuring top-level heartbeat detection, since you already have that working. ### What should happen When the parent flow run transitions to

CRASHED

due to pod/node loss, any non-terminal descendant runs created within that same execution context should also be reconciled to

CRASHED

. That includes: - the nested subflow run - any

RUNNING

/

PENDING

/ otherwise non-terminal task runs under that subflow ### What is happening instead Those descendant runs are being orphaned because: - the process vanished before they could write their final states - Prefect marked the top-level flow terminal - but did not fully propagate that crash to the in-flight descendants So this is really an orphaned child run state propagation issue. ### Important distinction You asked “failed or crashed” — for this case, crashed is the right state, not failed. ### Practical conclusion There is not a user-facing setting like: - “when parent flow crashes, crash all child task runs” If you are seeing this consistently in Prefect 3.x, I would treat it as: - either a product limitation in your current version - or a bug/regression ### Workarounds Until the behavior is fully reconciled automatically, the practical options are: 1. Periodic cleanup job Run a separate reconciliation process that: - finds non-terminal task/subflow runs - checks whether an ancestor flow run is

CRASHED

- force-sets those runs to

CRASHED

2. Separate infrastructure boundaries If a subflow is important enough to need independent lifecycle handling, run it as its own deployment/infrastructure boundary rather than in-process. Then it can be independently heartbeated and reconciled. 3. Upgrade/check current Prefect version This kind of descendant crash propagation behavior is version-sensitive. ### Best next step Given your very specific scenario, I’d suggest checking whether this is already a known issue. If not, it’s a good bug report:

“When a deployment flow run crashes due to pod/node loss, nested in-process subflow runs and task runs remain non-terminal instead of being transitioned to

CRASHED

.”

If you want, I can help you: - search for an existing GitHub issue/discussion, or - sketch a cleanup script that marks orphaned subflow/task runs as

CRASHED

.

Short answer: I can’t confirm from the available evidence that this is fixed in a released Prefect version yet. What I found: - I did not find a clear released issue/PR note saying: - “nested subflow/task runs are automatically marked

CRASHED

when the parent flow’s infrastructure dies” - I also did not find strong public issue/discussion coverage for your exact scenario - The docs/workaround pattern you mentioned — using a cleanup/reconciliation script for orphaned runs — is consistent with this not being fully solved universally So at the moment, the most honest answer is: Assume this is not reliably fixed unless you can verify it in your target version with a repro. A few useful signals: ### What suggests it may not be fully fixed - The documentation/workarounds still point people toward orphan cleanup patterns - I found no obvious release note in the latest changelog for descendant run crash propagation - I found no publicly indexed issue explicitly closed as fixed for: - in-process subflows - child task runs - same-pod infra loss / OOM / node teardown ### What I did find Some adjacent issues exist, but they are not the same thing: - run_deployment child flow errors not propagating to parent - Zombie flow automation false positives - wait_for dependency issue - older/adjacent state drift issues in 2.x, but not your exact 3.x descendant-orphan case Those tell us other users do run into state reconciliation edge cases, but they do not prove your specific bug is fixed. ### My recommendation Before upgrading solely for this, I would not assume a newer version fixes it. Instead: 1. Pick the exact version you’re considering 2. Reproduce with a minimal test: - parent deployment flow - in-process subflow - several tasks - kill pod / induce OOM 3. Confirm whether: - parent flow →

CRASHED

- subflow →

CRASHED

- child tasks →

CRASHED

If you want, I can help you build that repro. ### Practical takeaway Yes — the fact that docs suggest a cleanup script is a pretty strong hint that orphaned descendant runs are still something users may need to handle operationally. If helpful, I can next do one of two things: - help you write a minimal reproducible test for your current and target Prefect versions, or - help you write a cleanup script that marks orphaned subflow/task runs as

CRASHED

.