<@ULVA73B9P> I'm expecting an automation to trigge...
# ask-marvin
s
@Marvin I'm expecting an automation to trigger when all events are matched but this doesn't seem to happen, can you help me debug?
m
thought for 197.3 seconds
Absolutely—let’s track this down. In Prefect 3.x, “trigger when ALL events are matched” is done with a compound trigger that requires all child triggers to fire. The two most common reasons it doesn’t fire are: - The events aren’t being correlated to the same resource - The child triggers don’t fire within the compound trigger’s time window A few quick questions to pinpoint the issue: - Can you share the automation’s trigger config (JSON/YAML) or the automation name? Specifically: do you have a
type: compound
with
require: "all"
? - Do you intend “all events for the same resource”? If yes, are you using
for_each
(e.g.,
for_each: ["prefect.resource.id"]
) on each child event trigger? - What are your
within
values? The compound trigger’s
within
constrains how close in time the child firings must occur. While you gather that, here’s a concise debugging checklist: 1) Inspect the automation definition - Confirm it’s a compound trigger with
require: "all"
. - If you need “same resource” semantics, ensure each child event trigger includes the same
for_each
key(s), e.g.:
Copy code
{
  "type": "compound",
  "require": "all",
  "within": 3600,
  "triggers": [
    {
      "type": "event",
      "expect": [
        "prefect.flow-run.Running"
      ],
      "for_each": [
        "prefect.resource.id"
      ],
      "match": {
        "prefect.resource.id": "prefect.flow-run.*"
      }
    },
    {
      "type": "event",
      "expect": [
        "prefect.flow-run.Completed"
      ],
      "for_each": [
        "prefect.resource.id"
      ],
      "match": {
        "prefect.resource.id": "prefect.flow-run.*"
      }
    }
  ]
}
- If
for_each
is missing (or inconsistent) across children, “ALL” can be satisfied by events from different resources—often not what you want. CLI to inspect:
Copy code
prefect automations ls
prefect automations inspect --name "YOUR AUTOMATION" --json
Docs: Automations, Event triggers 2) Validate event names and match filters -
expect
must match event names exactly (case-sensitive) unless you use wildcards. -
match
and
match_related
are ANDed across labels. If you require a related resource, ensure it includes
prefect.resource.role
in the event. Example related labels must look like:
Copy code
"related": [
    {
      "prefect.resource.id": "<http://prefect.deployment.xxx|prefect.deployment.xxx>",
      "prefect.resource.role": "deployment"
    }
  ]
3) Check time windows - The compound trigger’s
within
means all child firings must occur within that window. If it’s short (e.g., 10s) and the child triggers fire 15s apart, the compound will not fire. - If child triggers are Proactive, they must have
within >= 10s
. 4) Stream events to verify what’s actually arriving - Confirm that the event payloads have the expected
event
,
resource.prefect.resource.id
, and any related roles/labels you match on:
Copy code
prefect events stream --format json
You can filter locally with
grep
or
jq
to see only the events you expect to match. 5) Confirm that the child triggers are firing - In the UI: Automations → your automation → check Activity/Executions for -
prefect.automation.triggered
(fired), - `prefect.automation.action.executed`/`failed` (actions), - absence of these indicates the compound never satisfied “ALL”. - If nothing is showing, it’s likely: - a `match`/`match_related` mismatch, - missing
for_each
to correlate the same resource, - or child firings fell outside the compound
within
window and were pruned. 6) Emit a test event (optional) - Useful to verify match logic quickly:
Copy code
prefect events emit "my.test.event" --resource-id "my.resource.id" --payload '{"test": true}'
Common culprits I see: - Missing
for_each
on all child event triggers when you need “same resource” - Very short
within
on the compound trigger -
expect
not exactly matching the event name
-
match_related
used but incoming events are missing
prefect.resource.role
on related resources If you can share: - The automation’s trigger JSON/YAML - An example of one event you expect to match (from
prefect events stream --format json
) I can pinpoint exactly why it’s not firing and suggest a precise fix. Helpful links: - Automations concept - Event triggers (require=all, within, for_each) - Events and event payloads - CLI: automations - CLI: events
s
@Marvin I should have mentioned this is for Prefect 2.20.16. Does that same apply?
m
thought for 164.2 seconds
Thanks for the version check — that helps. In Prefect 2.20.16, the “when all events are matched” behavior is implemented via a Compound trigger with
require: "all"
. Most of what I shared conceptually still applies, but there are a few 2.x specifics and CLI differences to watch for. What to verify in 2.20.16 - Trigger type: You should see
type: "compound"
with
require: "all"
. - “Same resource” correlation: Use
for_each
on each child event trigger (e.g.,
for_each: ["prefect.resource.id"]
) if you expect all events to refer to the same resource. If you omit or mismatch
for_each
across children, the automation can combine events from different resources and never satisfy your intent. - Time window: The compound trigger’s
within
requires all child firings to occur within that window. If it’s too short, the compound won’t fire even if all children fire, just not close enough in time. - Event names and filters:
expect
is case-sensitive and must match exactly (unless you use wildcards). `match`/`match_related` are ANDed across labels; related resources must include a
prefect.resource.role
for the match to succeed. - Posture constraints: Proactive child triggers require
within >= 10s
. Reactive can use
within: 0
. A minimal good example (2.20.16)
Copy code
{
  "type": "compound",
  "require": "all",
  "within": 3600,
  "triggers": [
    {
      "type": "event",
      "posture": "Reactive",
      "expect": [
        "prefect.flow-run.Running"
      ],
      "match": {
        "prefect.resource.id": "prefect.flow-run.*"
      },
      "for_each": [
        "prefect.resource.id"
      ]
    },
    {
      "type": "event",
      "posture": "Reactive",
      "expect": [
        "prefect.flow-run.Completed"
      ],
      "match": {
        "prefect.resource.id": "prefect.flow-run.*"
      },
      "for_each": [
        "prefect.resource.id"
      ]
    }
  ]
}
This only fires when the same flow run has emitted both Running and Completed within 1 hour. Quick debugging checklist (2.20.16) 1) Inspect the automation definition
Copy code
prefect automation ls
prefect automation inspect "YOUR AUTOMATION" --json
- Confirm
type: "compound"
,
require: "all"
, and that each child has the same
for_each
if you need “same resource”. 2) Stream events to see what’s actually arriving
Copy code
prefect event stream --format json
- Verify that the events you expect to match have: - The exact
event
name(s) in
expect
- Primary
resource.prefect.resource.id
and any
related
resources you filter on - For related resources,
prefect.resource.role
is present and correct 3) Check timing and thresholds - If the compound
within
is short, increase it. - If any child is Proactive, ensure
within >= 10
. 4) UI activity - In the Automations UI, open your automation and check the Activity/Executions log to see if child triggers are firing and whether the compound is evaluated but not satisfied. Common pitfalls I see in 2.20.16 - Missing or inconsistent
for_each
across child triggers when you mean “for the same resource”. -
expect
doesn’t exactly match the event names (case/typo). -
match_related
is set but incoming events don’t include
prefect.resource.role
in the related resources. - Compound
within
too small, so child firings don’t overlap. - Using a Proactive child trigger with
within < 10s
. If you can share: - The automation’s trigger JSON/YAML (or
prefect automation inspect --json
) - One or two example events you expect to match (grabbed from
prefect event stream --format json
) …I can pinpoint the exact mismatch and propose a precise fix. Useful references - Source (2.20.16) for compound/for_each behavior: - Automations schemas (2.20.16) - Trigger evaluation (2.20.16) - CLI help you can run locally:
Copy code
prefect automation --help
prefect event --help
Note: If you plan to upgrade later, Prefect 3.x keeps the same core ideas (compound
require: "all"
,
for_each
,
within
) but the CLI group for events is
prefect events
(plural) in 3.x, while it’s
prefect event
(singular) in 2.x.
s
@Marvin I have replaced the
eventSource
values with letters
A
-
F
. This is what the compound trigger looks like, can you see anything that may cause the trigger not to fire? I have checked all events and can confirm they all complete within 10 hours,
within
has a window of 12 hours.
m
thought for 87.9 seconds
Thanks for clarifying. Since you’re on Prefect 2.20.16, a compound trigger with require: "all" should work as you expect, but there are a few gotchas that commonly prevent firing. I don’t see the JSON/YAML for your trigger in your message—could you paste the compound trigger config (and, if possible, one example event you expect to match, redacted as needed)? That’ll let me spot the exact issue. In the meantime, here are the most likely culprits to check: Top things that block “ALL” from firing in 2.20.16 - A proactive child trigger: If any child is posture: "Proactive", it only fires on absence (after the window). If your events A–F all do occur, that proactive child will never fire and the compound won’t trigger. For “all events happened”, all children should be Reactive. - Matching on payload fields: Event matching only uses resource labels (
match
) and related resource labels (
match_related
), not payload. If
eventSource
is in the payload (e.g.,
payload.eventSource: "A"
), it will not be matched. Make sure the thing you’re matching (A–F) is either the event name in
expect
or a label on the primary/related resource. - for_each not aligned across children: If you intend “all events for the same resource,” each child event trigger must have the same
for_each
(e.g.,
["prefect.resource.id"]
). If one child has no
for_each
(or a different one), child firings won’t align and the compound won’t consider them “together.” - Time windows: The compound
within
(12h) constrains how close in time the child firings must occur. If any child’s own
within
or
after
logic delays or splits evaluation, the firings might not overlap within 12h. - Exact event names and filters:
expect
must exactly match (case-sensitive) unless you use a wildcard.
match_related
requires
prefect.resource.role
on related resources. Check
threshold
too—if a child has
threshold: 2
but you only see one event, it won’t fire. Quick checks you can run now - Inspect the automation:
Copy code
prefect automation inspect "YOUR AUTOMATION" --json
Confirm: -
type: "compound"
,
require: "all"
,
within: "12h"
(or seconds) - Every child has posture: "Reactive" for presence-of-event logic - Every child uses the same
for_each
if you need “same resource” semantics -
expect
values match actual event names;
threshold
is what you intend - Stream the events you expect to match and verify labels vs payload:
Copy code
prefect event stream --format json
Check that: - The event name is what your
expect
references (or pattern matches) - The values you match on live in resource/related labels (not payload) - Related resources include
prefect.resource.role
A minimal good pattern (2.20.16), if you want “A AND B for the same resource” - Ensure A/B are event names or resource labels (not payload) - Use the same
for_each
on both children:
Copy code
{
  "type": "compound",
  "require": "all",
  "within": 43200,  // 12 hours in seconds
  "triggers": [
    {
      "type": "event",
      "posture": "Reactive",
      "expect": ["my.event.A"],
      "match": {"prefect.resource.id": "my.resource.*"},
      "for_each": ["prefect.resource.id"],
      "threshold": 1
    },
    {
      "type": "event",
      "posture": "Reactive",
      "expect": ["my.event.B"],
      "match": {"prefect.resource.id": "my.resource.*"},
      "for_each": ["prefect.resource.id"],
      "threshold": 1
    }
  ]
}
If your A–F are not separate event names but a field (e.g., eventSource) inside the event payload, you have two options: - Move that field into a resource or related resource label at emit time, then match on it; or - Emit distinct event names for A–F and match via
expect
. If you paste: - The compound trigger JSON/YAML - One example event (from
prefect event stream --format json
) that should satisfy one child …I’ll pinpoint the exact mismatch and provide the precise fix. References (2.20.16 source, for the matching/compound logic)
- Automations schemas: https://github.com/PrefectHQ/prefect/blob/2.20.16/src/prefect/events/schemas/automations.py - Trigger evaluation: https://github.com/PrefectHQ/prefect/blob/2.20.16/src/prefect/server/events/triggers.py CLI help you can run
Copy code
prefect automation --help
prefect event --help
s
@Marvin eventSource value is in the format of
urn:service:productId:environment:objectType:objectName
, with
for_each: []
the same across all children.
expect
is the same across all children and each child has
posture: "Reactive"
m
thought for 77.3 seconds
Thanks for the extra detail. A few things jump out that commonly block a compound require: "all" from firing in Prefect 2.20.16, given your setup: Likely cause #1: Matching on payload instead of labels - Triggers only match on labels in: - resource (e.g.,
resource["prefect.resource.id"]
, custom labels you add there), and - related resources (via
match_related
, with
prefect.resource.role
set) - If your
eventSource
lives in the event payload (not in the resource/related labels), a child trigger with
match: {"eventSource": "urn:...A"}
will never match. - Fix options: - If you control the emitter, move
eventSource
into the resource labels (or make it the
prefect.resource.id
) when emitting:
Copy code
emit_event(
      event="your.event.name",
      resource={
        "prefect.resource.id": "urn:service:product:env:type:name",
        "eventSource": "urn:service:product:env:type:name",
        ...
      },
      payload={...}
    )
Then in the trigger child:
Copy code
"match": { "eventSource": "urn:service:...A" }
- Or, if your
prefect.resource.id
is already that URN, match that instead:
Copy code
"match": { "prefect.resource.id": "urn:service:...A" }
Likely cause #2: All children share the same expect, but only differ by eventSource - That’s fine, but only if the eventSource filter is applied to labels (see #1). If it’s in payload, all children will fail to match. Other checks that can silently block “ALL” - after usage: If any child has an
after
that didn’t occur in this 12h window, its bucket never starts, so the compound will never see that child as fired. - threshold: Ensure each child has
threshold: 1
(or the intended count). A higher threshold would require multiple events per child. - for_each: You said it’s
[]
for all children—this is acceptable. If you actually want “A–F for the same thing,” consider adding a shared label-based bucketing key, e.g.:
Copy code
"for_each": ["prefect.resource.id"]   // or a custom label like "product_id","environment","object_name"
Without
for_each
, any A + any B + ... anywhere in the workspace within 12h will satisfy the compound. How to confirm quickly 1) Inspect the automation
Copy code
prefect automation inspect "YOUR AUTOMATION" --json
- Verify each child has posture: "Reactive", the intended
threshold
, and the `match`/`match_related` you expect. 2) Stream a couple of your events and check where eventSource lives
Copy code
prefect event stream --format json
- Look for the A–F events and confirm: - Is
eventSource
present under
resource
(labels) or only under
payload
? - If you use
match_related
, do your related resources include
prefect.resource.role
? If you can paste: - The compound trigger JSON/YAML (redact values as needed), and - One example event (from
prefect event stream --format json
) that should match one child, …I can pinpoint the exact mismatch and provide a precise fix. References (2.20.16 source for behavior) - Automations schemas (compound/for_each): https://github.com/PrefectHQ/prefect/blob/2.20.16/src/prefect/events/schemas/automations.py - Trigger evaluation/matching: https://github.com/PrefectHQ/prefect/blob/2.20.16/src/prefect/server/events/triggers.py CLI help
Copy code
prefect automation --help
prefect event --help
s
@Marvin, I should note that automations are working using the same set up, it seems to be that this automation isn't being triggered by its events
m
thought for 74.2 seconds
Got it — since other automations with the same pattern are firing, this points to a subtle mismatch in this compound automation where at least one child never “fires,” so the compound
require: "all"
never completes. Here’s how I’d zero in on it quickly: Targeted checks for this automation - Match vs payload: If your
eventSource
is only in the event payload, child triggers with
match: {"eventSource": ...}
will never match. Triggers match only on labels in
resource
(and in
related
via
match_related
). If you can, put
eventSource
as a resource label (or make it the
prefect.resource.id
), then match on it:
Copy code
"match": { "prefect.resource.id": "urn:service:product:env:type:name-A" }
or
Copy code
"match": { "eventSource": "urn:service:product:env:type:name-A" }  // only if it’s a resource label
- match vs match_related: If
eventSource
is actually on a related resource, it must be matched under
match_related
, and the related object must include
prefect.resource.role
in the incoming events. Otherwise no match. - Exact event name: All children share the same
expect
. Confirm the event name in the stream exactly matches
expect
(case-sensitive) or use a wildcard pattern that actually matches. - Thresholds: Ensure each child has
threshold: 1
(or what you intend). A higher threshold would block that child from firing. - after on any child: If any child has
after
, its bucket won’t start until the
after
event arrives; if that doesn’t occur in your 12h window, the child won’t fire. - for_each: You said it’s
[]
for all children — that’s valid. If you actually mean “A–F must all occur for the same thing,” add a consistent label for_each (e.g.,
["prefect.resource.id"]
or labels that identify the same product/env/object). - Time windows: With
within: 12h
on the compound, all child firings must occur within 12h of each other. If the earliest child fired slightly more than 12h before the last one, it will be pruned during evaluation. Quick way to identify the missing child - Temporarily switch the compound to
require: "any"
to confirm that at least one child fires. Or, clone this automation into six single-child automations (one per letter A–F). The one(s) that never fire isolate the problem. - Or keep it compound and loosen one child’s match to confirm it can fire; then tighten it back. What to run right now - Inspect the automation in full to confirm each child’s config (posture, expect, match/match_related, threshold, after, for_each):
Copy code
prefect automation inspect "YOUR AUTOMATION" --json
- Stream the actual events and verify where
eventSource
lives and that the event name matches `expect`:
Copy code
prefect event stream --format json
Check a sample event you expect to match: - Does it have
resource.prefect.resource.id
and any custom labels you match on? - If you use
match_related
, do related resources in the event include
prefect.resource.role
? - Is
eventSource
in resource labels (good) or only in payload (won’t match)? If you share: - The compound trigger JSON/YAML (redacted), and - One example of an A–F event from
prefect event stream --format json
…I can pinpoint exactly which child isn’t matching and why. Helpful references (Prefect 2.20.16 source for this behavior) - Automations schemas (compound/for_each): https://github.com/PrefectHQ/prefect/blob/2.20.16/src/prefect/events/schemas/automations.py - Trigger evaluation/matching logic: https://github.com/PrefectHQ/prefect/blob/2.20.16/src/prefect/server/events/triggers.py CLI help
Copy code
prefect automation --help
prefect event --help
s
@Marvin this is what the event looks like that should trigger the automation
m
thought for 66.5 seconds
I don’t see the event JSON in your message—could you paste one full event (redact values as needed) exactly as received by Prefect? Ideally copy from:
Copy code
prefect event stream --format json
Include at least: - event - occurred - resource (all labels) - related (with roles) - payload What I’ll check immediately: - event exactly matches your children’s
expect
(case-sensitive; or wildcard patterns really match) - eventSource location: is it under resource labels (good), related labels (then use match_related), or only in payload (won’t match)? - If using match_related, each related item must include
prefect.resource.role
- Any child has
after
set (can prevent its bucket from starting) - threshold on each child (should be 1 unless you intend more) Two common gotchas given your description: 1) eventSource only in payload - Triggers match only on labels in resource/match or related/match_related, not payload. - Fix: put eventSource into resource labels (or make it the
prefect.resource.id
) when emitting and match that label; or emit distinct event names A–F and match via
expect
. 2) match_related missing roles - If you’re matching via related, ensure incoming events include
prefect.resource.role
on the related object you’re filtering; otherwise the child never fires. If you’d like to isolate which child isn’t firing: - Temporarily clone this automation into six single-child automations (A–F). The one(s) that never fire will reveal the mismatch quickly. - Or temporarily set the compound
require
to "any" to confirm at least one child fires, then change back to "all". Quick sanity checks
Copy code
prefect automation inspect "YOUR AUTOMATION" --json
prefect event stream --format json
Happy to pinpoint the exact issue once I can see one sample event and (optionally) your compound trigger JSON/YAML.