https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have this event set up on my self h...
# ask-marvin
n
@Marvin I have this event set up on my self hosted prefect server which should detect zombie flows:
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.flow-run.*"
    ]
  },
  "match_related": {},
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Completed",
    "prefect.flow-run.heartbeat",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 300
}
All flows have the env variables 
PREFECT_FLOWS_HEARTBEAT_FREQUENCY
set to 30. However, this is not working. Why?
m
✅ thought for 123.5 seconds
Most likely: your self-hosted server version does not support flow-run heartbeats / zombie-flow detection the way your automation expects, or the flow runs are not actually emitting heartbeat events. A few important points: 1. Flow heartbeats are a newer Prefect 3 feature 
PREFECT_FLOWS_HEARTBEAT_FREQUENCY
is only meaningful in newer Prefect 3 releases. Since I know you’ve mentioned using Prefect 3.4.9 before, that immediately stands out: that version is older than the versions where flow-run heartbeat support was introduced/fixed. If your workers/flow runtime are on 3.4.9, setting the env var alone won’t make 
prefect.flow-run.heartbeat
events appear. Docs: Detect zombie flows 2. Heartbeats are emitted by the flow runtime, not just the server Your automation only works if the running flow process is actually sending 
prefect.flow-run.heartbeat
events to the API. So all of these must be true: - the flow runtime version supports heartbeats - the flow is a deployment-backed run - the runtime has 
PREFECT_API_URL
pointed at your self-hosted server - events are enabled/reaching the server 3. Self-hosted server does support event triggers So “self-hosted” by itself is not the reason. Event triggers are supported in OSS/server. The more likely issue is no heartbeat events are being emitted. Event trigger docs: Event triggers 4. Your trigger shape is generally reasonable This pattern is the standard idea for zombie detection: - `after`: first heartbeat starts the window - `expect`: another heartbeat or terminal state must arrive within 
within
- `posture: Proactive`: fire when expected events do not arrive With heartbeat frequency 30s and 
within: 300
, that should be plenty. What I’d check first Check whether heartbeat events exist at all In the UI, inspect events for a running flow run and see whether you ever see 
prefect.flow-run.heartbeat
. If you do not see them, the automation can never fire. Check your Prefect version on the runtime side From the environment actually running the flow:
Copy code
bash
prefect version
If that’s still 3.4.9, I would strongly suspect that’s the problem. Check the env var is present in the flow execution environment Not just where you build/deploy, but where the flow process actually runs.
Copy code
bash
echo $PREFECT_FLOWS_HEARTBEAT_FREQUENCY
echo $PREFECT_API_URL
Check the run is a deployed flow run Zombie detection is intended for deployed flow runs; ad hoc local execution may not behave the same way. Bottom line The most likely answer is: 
PREFECT_FLOWS_HEARTBEAT_FREQUENCY=30
is not enough on Prefect 3.4.9, because that version likely predates the flow heartbeat/zombie-detection support you’re trying to use. So the fix is usually to upgrade both the server and the flow runtime/worker images to a newer Prefect 3 version, then verify that 
prefect.flow-run.heartbeat
events are actually appearing. If you want, I can help you narrow this down quickly with a 3-step check: 1. your server version 2. your worker/runtime version 3. whether 
prefect.flow-run.heartbeat
appears in Events for an active run
n
@Marvin Both server and flow runs use Prefect 3.6.12. All flows are deployed. Events do reach the server because I see logs and other traces. PREFECT_API_URL is set
m
✅ thought for 263.7 seconds
Thanks — that changes the answer. For Prefect 3.6.12, the issue is very likely that you’re setting the wrong heartbeat setting for that version. In 3.6.12, flow-run heartbeat emission still used the runner heartbeat setting, not the newer flows setting. So this:
Copy code
bash
PREFECT_FLOWS_HEARTBEAT_FREQUENCY=30
is likely not enough in 3.6.12. You should set:
Copy code
bash
PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30
on the environment where the flow run is actually executed. Why this matters In 3.6.12, 
prefect.flow-run.heartbeat
events existed, and proactive triggers existed too, but heartbeats were still wired through the runner-side configuration. The later 
PREFECT_FLOWS_HEARTBEAT_FREQUENCY
behavior/documentation landed after that. So your automation can be perfectly valid, events/traces can generally work, and 
PREFECT_API_URL
can be correct — but if heartbeat emission itself is disabled, you’ll never get the 
after
event that starts the proactive window. What to check 1. Confirm heartbeat events are present Look for 
prefect.flow-run.heartbeat
in Events for an actively running flow run. If none appear, the automation won’t work. 2. Set the old setting for 3.6.12 In the runtime environment:
Copy code
bash
   PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30
3. Optionally set both during transition To avoid version confusion across images/components:
Copy code
bash
   PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30
   PREFECT_FLOWS_HEARTBEAT_FREQUENCY=30
4. Restart workers / runtime containers So the setting is actually picked up. Your trigger itself Your trigger JSON looks conceptually fine for zombie detection:
Copy code
json
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.flow-run.*"
    ]
  },
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Completed",
    "prefect.flow-run.heartbeat",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 300
}
So I would not start by changing the automation logic. I’d first verify that heartbeat events are emitted at all. Relevant docs - Detect zombie flows - Event triggers If you want, I can also help you with a very quick test flow to confirm whether 
prefect.flow-run.heartbeat
is being emitted in your 3.6.12 setup.
n
@Marvin How can I change the automation such that it works with state TYPE instead of state NAMES? The idea is that, if a flow run completes with a state named SomeStates of type COMPLETED, it should not trigger a CrashedZombie state.
m
✅ thought for 121.8 seconds
Yes — in Prefect 3.x you should filter on the event resource labels, not just the event name, if you care about state type. The key distinction is: - 
expect
matches the event name - 
match
matches event/resource labels like 
prefect.state-type
and 
prefect.state-name
For flow-run state events, Prefect includes both: - 
prefect.state-name
- 
prefect.state-type
So if a flow finishes with: - state name = 
SomeStates
- state type = 
COMPLETED
then the event name is based on the name, but the event resource still has:
Copy code
text
prefect.state-type = COMPLETED
So to avoid treating custom completed states like failures/crashes, configure the automation to check:
Copy code
text
prefect.state-type
instead of relying on state names. Example trigger pattern:
Copy code
json
{
  "match": {
    "prefect.resource.id": "prefect.flow-run.*",
    "prefect.state-type": "CRASHED"
  },
  "expect": [
    "prefect.flow-run.*"
  ]
}
If your current automation is keying off a list of names, change it so the filter uses:
Copy code
text
prefect.state-type: CRASHED
or whatever type you actually want to detect. If you want to exclude custom state names as well, you can also add a state-name filter, but for your case the cleaner fix is to use 
prefect.state-type
. Useful docs: - Event triggers - Flow run event payloads If you paste your current automation YAML/JSON, I can rewrite it to use 
state-type
directly.
n
@Marvin Please change to automation provided early such that it expects state types instead of state names
m
✅ thought for 5.6 seconds
Sure — I can do that, but I need the actual automation definition you want edited. Please paste the earlier automation YAML/JSON here, and I’ll convert it from matching state names to matching state types. In general, the change usually looks like this: - remove name-based matching like 
prefect.state-name
- replace it with type-based matching like 
prefect.state-type
- avoid relying on 
expect
values that encode custom state names when you really mean canonical types For example, this kind of name-based logic:
Copy code
json
{
  "match": {
    "prefect.resource.id": "prefect.flow-run.*",
    "prefect.state-name": "CrashedZombie"
  }
}
would become:
Copy code
json
{
  "match": {
    "prefect.resource.id": "prefect.flow-run.*",
    "prefect.state-type": "CRASHED"
  }
}
Send over the original automation and I’ll rewrite it exactly.
n
@Marvin
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.flow-run.*"
    ]
  },
  "match_related": {},
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Completed",
    "prefect.flow-run.heartbeat",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 300
}
m
✅ thought for 25.7 seconds
You can’t directly make 
expect
use state types — 
expect
matches event names, and flow-run event names are built from the state name. So for your current automation:
Copy code
json
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.flow-run.*"
    ]
  },
  "match_related": {},
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.Failed",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Completed",
    "prefect.flow-run.heartbeat",
    "prefect.flow-run.Crashed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 300
}
the problem is that a custom completed state like 
SomeStates
emits an event like:
Copy code
text
prefect.flow-run.SomeStates
not 
prefect.flow-run.Completed
. So if you want this automation to stop treating custom 
COMPLETED
states as missing terminal states, you have two options: Option 1: broaden 
expect
to all flow-run state events and use `match`/logic elsewhere Option 2: use a different automation pattern, because proactive 
expect
is inherently name-based A minimal rewrite that is safer is:
Copy code
json
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.flow-run.*"
    ]
  },
  "match_related": {},
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.*",
    "prefect.flow-run.heartbeat"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 300
}
But important caveat: this now expects any flow-run event, not specifically terminal state types. So if your goal is specifically: - heartbeat happens - then within 300s, accept any state whose type is 
COMPLETED
/ 
FAILED
/ 
CANCELLED
/ 
CRASHED
- and only alert otherwise then this exact proactive automation model can’t express that purely with 
state-type
in 
expect
, because 
expect
is event-name-based. So the honest answer is: there is no direct “convert 
expect
from state names to state types” version of this automation. Why: - 
expect
uses event names like 
prefect.flow-run.Completed
- custom state names produce custom event names - 
prefect.state-type
is available in 
match
, but not as a type-aware substitute for 
expect
Docs: - Event triggers - Flow run events If you want, I can help you redesign this automation so it still detects zombie/crashed runs without being broken by custom completed state names.
n
@Marvin Can I not simply use 
prefect.state-type.COMPLETED
etc inside the expect block?
m
✅ thought for 7.4 seconds
No — not in 
expect
. 
expect
only matches event names, and 
prefect.state-type.COMPLETED
is not an event name. So these are valid in `expect`:
Copy code
json
"expect": [
  "prefect.flow-run.Completed",
  "prefect.flow-run.Failed",
  "prefect.flow-run.Crashed"
]
but this is not valid:
Copy code
json
"expect": [
  "prefect.state-type.COMPLETED"
]
because 
prefect.state-type
is a resource label on the event payload, not an event name. For flow-run state events, Prefect emits something conceptually like:
Copy code
json
{
  "event": "prefect.flow-run.SomeStates",
  "resource": {
    "prefect.resource.id": "prefect.flow-run.<id>",
    "prefect.state-name": "SomeStates",
    "prefect.state-type": "COMPLETED"
  }
}
So: - 
event
/ 
expect
-> 
prefect.flow-run.SomeStates
- 
match
-> can inspect 
prefect.state-type = COMPLETED
That’s why custom state names break name-based 
expect
matching. References: - Event triggers - Flow run events If you want, I can suggest the closest workable replacement for your zombie-detection automation.
3 Views
Open in Slack
PreviousNext
Powered by