hey <@ULVA73B9P> I have the <zombie flow process r...
# ask-marvin
p
hey @Marvin I have the zombie flow process running with the only change being that sends a slack message afterwards. I've noticed that the process has some false positives with my really short running tasks - they are usually between 30s - 1 minute. It doesn't mark the flow as crashed (or at least there is nothing in the logs and the status hasn't changed) but it still sends the slack message. Here's a screen shot of one that triggered this. Any ideas about how to resolve? Maybe I add another condition that the flow must have been up for at least a minute? Or change the heartbeat frequency/detection to avoid this?
m
thought for 498.4 seconds
Thanks for the details — this is a common tuning issue with the zombie-flow recipe, especially for short-lived runs. What’s likely happening - If your automation starts “watching” right after Running and your flows finish in 30–60s, you can hit a race: the automation fires because it didn’t see a heartbeat “in time,” but by the time actions run the flow has already completed. If your Slack is attached to that same automation, you’ll see a message even though the run never actually crashed (and change-state may no-op). - Short runs may complete before the first heartbeat, depending on your heartbeat settings. If detection is keyed off Running (instead of the first Heartbeat), you’ll see more of these false positives. What to change to reduce false positives 1) Gate detection on the first heartbeat, not on Running - Configure the trigger to start “after” a flow-run Heartbeat event and “expect” either another Heartbeat or a terminal state (Completed/Failed/Cancelled/Crashed/TimedOut) within a window. - This avoids alerting for runs that complete before the first heartbeat. - Docs: Detect zombie flows, Event triggers 2) Increase the detection window (within) or require multiple misses - If your runner heartbeat is every 30s, set the proactive trigger window to at least 90–120s. This ensures short runs have time to complete and avoids alerting on transient delays. - If you’ve got a “threshold” setting available in your trigger, bump it (e.g., require two missed intervals) to be more conservative. 3) Separate state change from notification - Best practice: create two automations - Automation A (Zombie detection): proactive trigger (after Heartbeat), within 90–120s, action = ChangeFlowRunState to Crashed. - Automation B (Notification): simple event trigger on flow-run.Crashed that sends Slack. - This way you only get Slack when the run is actually marked Crashed, eliminating “notify-but-not-crashed” cases. 4) Ensure heartbeats are actually enabled on your runners - In Prefect 3.x, heartbeats are emitted by the runner infrastructure, controlled by
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
(min 30s). If unset, heartbeats may not be emitted and detection based on heartbeats won’t behave as expected. - Set this on the runner/worker environment (or profile) and pick a window that’s ~3–4× that frequency. - Reference: runner settings show
heartbeat_frequency
with a minimum of 30s. 5) Optional: exclude very short-lived flows - If you have a specific set of deployments that routinely finish in under a minute, consider filtering the automation by deployment name or tags so you don’t monitor those with the same sensitivity. A conservative configuration that works well for short flows - Runner heartbeat: 30s - Trigger “after”: flow-run Heartbeat - Expect: Heartbeat OR Completed/Failed/Cancelled/Crashed/TimedOut - Posture: Proactive - Within: 120s - for_each: flow-run (so each run is evaluated independently) - Actions: ChangeFlowRunState to Crashed (in automation A), and a separate automation to send Slack on flow-run.Crashed Helpful docs - Detect zombie flows: Recipe - Event triggers and proactive windows: Concepts Could you share: - A screenshot or JSON of your current automation (especially the trigger’s “after/expect/within/threshold” and the actions list)? - Whether
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
is set on your runner/worker (and to what value)? With that, I can suggest exact values or an updated trigger config to stop the false positives.
p
ah! two different events. That's the way to do it! Thank you