Thanks for the details — this is a common tuning issue with the zombie-flow recipe, especially for short-lived runs.
What’s likely happening
- If your automation starts “watching” right after Running and your flows finish in 30–60s, you can hit a race: the automation fires because it didn’t see a heartbeat “in time,” but by the time actions run the flow has already completed. If your Slack is attached to that same automation, you’ll see a message even though the run never actually crashed (and change-state may no-op).
- Short runs may complete before the first heartbeat, depending on your heartbeat settings. If detection is keyed off Running (instead of the first Heartbeat), you’ll see more of these false positives.
What to change to reduce false positives
1) Gate detection on the first heartbeat, not on Running
- Configure the trigger to start “after” a flow-run Heartbeat event and “expect” either another Heartbeat or a terminal state (Completed/Failed/Cancelled/Crashed/TimedOut) within a window.
- This avoids alerting for runs that complete before the first heartbeat.
- Docs:
Detect zombie flows,
Event triggers
2) Increase the detection window (within) or require multiple misses
- If your runner heartbeat is every 30s, set the proactive trigger window to at least 90–120s. This ensures short runs have time to complete and avoids alerting on transient delays.
- If you’ve got a “threshold” setting available in your trigger, bump it (e.g., require two missed intervals) to be more conservative.
3) Separate state change from notification
- Best practice: create two automations
- Automation A (Zombie detection): proactive trigger (after Heartbeat), within 90–120s, action = ChangeFlowRunState to Crashed.
- Automation B (Notification): simple event trigger on flow-run.Crashed that sends Slack.
- This way you only get Slack when the run is actually marked Crashed, eliminating “notify-but-not-crashed” cases.
4) Ensure heartbeats are actually enabled on your runners
- In Prefect 3.x, heartbeats are emitted by the runner infrastructure, controlled by
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
(min 30s). If unset, heartbeats may not be emitted and detection based on heartbeats won’t behave as expected.
- Set this on the runner/worker environment (or profile) and pick a window that’s ~3–4× that frequency.
- Reference: runner settings show
heartbeat_frequency
with a minimum of 30s.
5) Optional: exclude very short-lived flows
- If you have a specific set of deployments that routinely finish in under a minute, consider filtering the automation by deployment name or tags so you don’t monitor those with the same sensitivity.
A conservative configuration that works well for short flows
- Runner heartbeat: 30s
- Trigger “after”: flow-run Heartbeat
- Expect: Heartbeat OR Completed/Failed/Cancelled/Crashed/TimedOut
- Posture: Proactive
- Within: 120s
- for_each: flow-run (so each run is evaluated independently)
- Actions: ChangeFlowRunState to Crashed (in automation A), and a separate automation to send Slack on flow-run.Crashed
Helpful docs
- Detect zombie flows:
Recipe
- Event triggers and proactive windows:
Concepts
Could you share:
- A screenshot or JSON of your current automation (especially the trigger’s “after/expect/within/threshold” and the actions list)?
- Whether
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
is set on your runner/worker (and to what value)?
With that, I can suggest exact values or an updated trigger config to stop the false positives.