<@ULVA73B9P> currently i am doing batch streaming ...
# ask-marvin
k
@Marvin currently i am doing batch streaming in spark, on a completed batch process i am emiting the event:
Copy code
{
  "id": "0199276c-2743-7107-8aef-532bf5b0e0c2",
  "event": "axos.bronze.invoice_lines.updated",
  "occurred": "2025-09-08T03:43:43.423Z",
  "payload": {
    "table_name": "invoice_lines",
    "record_count": 199997,
    "timestamp": "2025-09-08T15:43:43.403998"
  },
  "received": "2025-09-08T03:43:43.500Z",
  "related": [],
  "resource": {
    "prefect.resource.id": "bronze.invoice_lines",
    "prefect.resource.name": "Bronze Table: invoice_lines"
  }
}
i also have set up a workpool which has a deployment attached, i can manually run these deployments and they run fine. to automatically trigger this deployment i have set up a a "Automation" called bronze to silver automation which runs the deployment that works. the trigger config is as follows: { "type": "event", "match": { "prefect.resource.id": "bronze.*" }, "match_related": [], "after": [], "expect": [ "axos.bronze.*.updated" ], "for_each": [ "prefect.resource.id" ], "posture": "Reactive", "threshold": 1, "within": 0 } and the action is run a deployment bronze-to-silver-pipeline > broze-to-silver-auto (the working deployment), why is this not running when i can see the emited event in the UI
m
thought for 94.3 seconds
Thanks for the detail — this helps a lot. The issue is almost certainly the field paths in your trigger. In Prefect 3, the event’s resource attributes live under the
resource
object. Your emitted event shows:
Copy code
"resource": {
  "prefect.resource.id": "bronze.invoice_lines",
  "prefect.resource.name": "Bronze Table: invoice_lines"
}
So your trigger should reference
resource.prefect.resource.id
(not
prefect.resource.id
at the top level). The same applies to your
for_each
. Try this trigger config:
Copy code
{
  "type": "event",
  "match": {
    "resource.prefect.resource.id": "bronze.*"
  },
  "match_related": [],
  "after": [],
  "expect": [
    "axos.bronze.*.updated"
  ],
  "for_each": [
    "resource.prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1
}
Notes and quick checks: - You can omit
within
when
threshold
is 1;
within: 0
isn’t needed. - Make sure the automation is enabled. Automations only react to events that arrive after they’re enabled. - Double‑check the deployment selected in the action (I noticed “broze” vs “bronze” in the name in your message—just ensure it points to the correct, working deployment). - Since manual runs work, your worker/pool is fine; this is a matching problem. How to confirm fast: - In the UI, open your Automation and use the Test/Evaluate feature with the exact event JSON you pasted. You should see it match when the path is
resource.prefect.resource.id
. - Check the automation’s evaluation/history panel for “why didn’t this fire?” feedback on the specific event. Docs for reference: - Event triggers (concepts) - Create an automation to run a deployment If you update the paths and it still doesn’t fire, paste a screenshot of the automation’s evaluation result (or the Test output) and I’ll help you zero in further.
k
when you mean open the UI and quickly test can you explain futher, i cant seem to find where im supposed to be looking
@Marvin when you mean open the UI and quickly test can you explain futher, i cant seem to find where im supposed to be looking
m
thought for 89.9 seconds
Totally—here’s where to look and how to “test” it in the UI. Assuming Prefect 3.x (Cloud or Server): 1) Open your automation detail - Left nav: Automations - Click your “bronze to silver automation” - Confirm the toggle at the top says Enabled 2) Check the trigger config and edit if needed - In the automation detail, expand the Trigger section and click Edit - Switch to the Advanced/JSON editor if you’re using JSON - Update the matcher paths to use resource-prefixed keys (example below), then Save
Copy code
{
  "type": "event",
  "match": {
    "resource.prefect.resource.id": "bronze.*"
  },
  "expect": [
    "axos.bronze.*.updated"
  ],
  "for_each": [
    "resource.prefect.resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1
}
3) Send a fresh event and watch the automation history Important: Automations only evaluate events that arrive after they’re enabled/saved. - Re‑emit your event (or wait for the next one) - In the automation detail page, scroll to the bottom section that shows recent evaluations/runs (it may be labeled History, Evaluations, or Automation runs). You should see either: - Triggered → Action submitted (then your deployment run), or - Not triggered with a short reason (e.g., field didn’t match) 4) Cross-check the event as Prefect received it - Left nav: Events - Find your event (filter by event name axos.bronze.*.updated or time) - Click it and review the JSON Prefect ingested; confirm the fields are under the resource object (e.g., resource.prefect.resource.id = bronze.invoice_lines) - This is the source of truth for the exact field paths your trigger must match If you don’t see an explicit “Test” or “Evaluate” button in your UI build, use the flow above: save the trigger, re-emit an event, then check the automation’s History/Evaluations. That’s the most reliable way to validate matching. Quick sanity checks: - Enablement: Make sure the automation is Enabled before you re-emit the event - Deployment action: Verify you selected the correct deployment (watch for typos like “broze” vs “bronze”) - Worker availability: Since manual runs work, you’re likely fine, but you can also check the deployment’s Run history to confirm action→flow run creation - Matching paths: Use resource.prefect.resource.id, not top-level prefect.resource.id Docs: - Event triggers (concepts) - Create an automation to run a deployment If you’d like, paste a screenshot of your Automation detail page (Trigger JSON and the History section) and the specific event from the Events page, and I’ll pinpoint why it isn’t matching.
k
@Marvin i have pasted you 3 images, of my automation, its config and an event that should of been captured and triggered, however even with the update, my automation is not triggering (this is a new event not the original one i pasted)
m
thought for 111.8 seconds
Thanks for the update — I can’t see images in Slack, so could you paste the raw JSON of: - The Automation config (the JSON block from the Advanced editor) - The exact event JSON from the Events page (click the event and copy the raw JSON) That’ll let me spot the mismatch precisely. In the meantime, here’s the most likely culprit and a quick way to prove it: What’s likely happening - The resource attribute key in your event is literally "prefect.resource.id". When you write a match like
resource.prefect.resource.id
, the engine interprets that as nested objects (resource → prefect → resource → id), which do not exist in your event. So it never matches. Fast test to isolate matching vs. action - Temporarily remove the match and for_each and trigger only on the event name. If this fires, we know the issue is the match path. Example:
Copy code
{
  "type": "event",
  "match": {},
  "match_related": [],
  "after": [],
  "expect": [
    "axos.bronze.*.updated"
  ],
  "posture": "Reactive",
  "threshold": 1
}
- Save (ensure the automation is Enabled), emit a fresh event, and check the automation’s History/Evaluations section on the Automation detail page. You should see either Triggered or a “did not trigger” reason. If that works, refine without dotted keys Because your payload already carries the table, you can avoid the dotted resource key entirely and still get per-table behavior:
Copy code
{
  "type": "event",
  "match": {},
  "expect": [
    "axos.bronze.*.updated"
  ],
  "for_each": [
    "payload.table_name"
  ],
  "posture": "Reactive",
  "threshold": 1
}
This will run once per table_name on those events. If you must match the resource id The key name contains dots, which is tricky. Try one of these path forms (they depend on your UI/build; use one at a time): - Bracketed key:
Copy code
"match": {
  "resource[\"prefect.resource.id\"]": "bronze.*"
},
"for_each": [
  "resource[\"prefect.resource.id\"]"
]
If that doesn’t work, fall back to the no-match approach above, or add a simpler key to your produced event (e.g., also include
"id": "bronze.invoice_lines"
under
resource
), then match
resource.id
. Where to look in the UI - Left nav: Automations → open your automation → ensure Enabled - Edit → Advanced (JSON) to paste the exact config - After saving, emit a new event - On the same automation page, scroll to the bottom panel that shows evaluations/history for the most recent events; it will show “Triggered” or why it didn’t fire - Left nav: Events → open the event → copy the raw JSON to confirm exact field paths CLI option to see events (useful to confirm the exact shape)
Copy code
prefect events stream --format json
Docs - Event triggers: https://docs-3.prefect.io/v3/concepts/event-triggers - Create an automation to run a deployment: https://docs-3.prefect.io/v3/how-to-guides/automations/creating-deployment-triggers If you paste the two JSON blobs (automation + event), I’ll give you the exact working match/for_each paths.
k
@Marvin here is the Event feed raw payload:
Copy code
{
  "id": "0199279e-aa32-7fe5-950d-3e909139ef4e",
  "event": "axos.bronze.invoice_line_items.updated",
  "occurred": "2025-09-08T04:38:53.743Z",
  "payload": {
    "table_name": "invoice_line_items",
    "record_count": 199997,
    "timestamp": "2025-09-08T16:38:53.734297"
  },
  "received": "2025-09-08T04:38:53.774Z",
  "related": [],
  "resource": {
    "prefect.resource.id": "bronze.invoice_line_items",
    "prefect.resource.name": "Bronze Table: invoice_line_items"
  }
}
and here is the automation trigger in JSON: { "type": "event", "match": { "resource.prefect.resource.id": "bronze.*" }, "match_related": {}, "after": [], "expect": [ "axos.bronze.*.updated" ], "for_each": [ "resource.prefect.resource.id" ], "posture": "Reactive", "threshold": 1, "within": 0 } what needs to change, is it the format of my emmited event, or is it the json config or is it something else
m
thought for 46.2 seconds
Thanks for sharing the raw JSON — that makes it clear. What’s happening - Your event has a literal key with dots: resource["prefect.resource.id"]. - In automation triggers, dot notation is used for nested paths. So
resource.prefect.resource.id
is interpreted as resource → prefect → resource → id, which does not exist in your event. That’s why it never matches. You have two solid paths forward: Option A (no event changes): Match and fan-out using payload.table_name This avoids the dotted key entirely and still gives you per-table behavior. Use this trigger:
Copy code
{
  "type": "event",
  "match": {},
  "expect": [
    "axos.bronze.*.updated"
  ],
  "for_each": [
    "payload.table_name"
  ],
  "posture": "Reactive",
  "threshold": 1
}
Notes: - Omit within (not needed when threshold = 1). - Set match_related to [] or omit it (empty object isn’t needed). Option B (adjust your emitted event): Add simple aliases under resource If you prefer matching on the resource, emit these additional fields alongside your current ones:
Copy code
"resource": {
  "id": "bronze.invoice_line_items",
  "name": "Bronze Table: invoice_line_items",
  "prefect.resource.id": "bronze.invoice_line_items",
  "prefect.resource.name": "Bronze Table: invoice_line_items"
}
Then your trigger can use simple paths:
Copy code
{
  "type": "event",
  "match": {
    "resource.id": "bronze.*"
  },
  "expect": [
    "axos.bronze.*.updated"
  ],
  "for_each": [
    "resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1
}
Why not escape the dotted key? - Prefect’s trigger path matching uses dot-path traversal; there isn’t documented support to “escape” dots in a key name within match/for_each paths. Using payload.table_name or adding alias keys is the reliable approach. After you update: 1) Save the automation and ensure it’s Enabled. 2) Emit a fresh event (automations only evaluate new events). 3) On the Automation detail page, check the History/Evaluations section to confirm it triggered. 4) Also verify the event in Events to confirm the exact fields. References: - Event triggers (concepts) - Create an automation to run a deployment If you want me to tailor the action to pass parameters (e.g., use the table name in your deployment parameters), share your deployment’s parameter schema and I’ll give you the exact action template to use.
k
@Marvin i dont mind how simple this gets, just as long as it runs on the event, from looking at it my event should trigger the automation, this is the current event config: { "type": "event", "match": { "resource.id": "bronze.*" }, "match_related": {}, "after": [], "expect": [ "axos.bronze.*.updated" ], "for_each": [ "resource.id" ], "posture": "Reactive", "threshold": 1, "within": 0 } and this event: Event Feed Axos bronze invoice lines updated • DetailsRaw
Copy code
{
  "id": "019927be-6b6b-71b5-99ad-d5847c8e62d0",
  "event": "axos.bronze.invoice_lines.updated",
  "occurred": "2025-09-08T05:13:34.825Z",
  "payload": {
    "table_name": "invoice_lines",
    "record_count": 33623,
    "timestamp": "2025-09-08T17:13:34.820654"
  },
  "received": "2025-09-08T05:13:34.844Z",
  "related": [],
  "resource": {
    "id": "bronze.invoice_lines",
    "name": "Bronze Table: invoice_lines",
    "prefect.resource.id": "bronze.invoice_lines",
    "prefect.resource.name": "Bronze Table: invoice_lines"
  }
}
tell me why it was not picked up and what needs to change, i am sure it didnt run, can i make my event simpler?
m
thought for 37.2 seconds
Thanks — your event shape looks good now. The reason it didn’t fire is your trigger’s match_related is set to an object
{}
. That effectively requires at least one related event to exist and match (but your event has
"related": []
), so the trigger never evaluates as true. Two fixes (pick one): 1) Minimal working config (keep your resource.id matching)
Copy code
{
  "type": "event",
  "expect": [
    "axos.bronze.*.updated"
  ],
  "match": {
    "resource.id": "bronze.*"
  },
  "for_each": [
    "resource.id"
  ],
  "posture": "Reactive",
  "threshold": 1
}
Notes: - Remove match_related (or set it to an empty list
[]
, not
{}
). - Remove within when threshold = 1 (it’s unnecessary). 2) Simplest possible config (no match/for_each; just run on any bronze updated event)
Copy code
{
  "type": "event",
  "expect": [
    "axos.bronze.*.updated"
  ],
  "posture": "Reactive",
  "threshold": 1
}
After updating: - Save the automation and ensure it’s Enabled. - Emit a fresh event (automations only react to new events). - Open the automation’s page and check the History/Evaluations section to confirm it triggered. - Also verify the action points to the exact deployment you expect. Your event is already “simple enough” — no need to change it further. If it still doesn’t fire after using one of the JSONs above, grab the latest entry from the automation’s History/Evaluations panel; it will usually state the reason it did not trigger (e.g., “no related events matched”). Share that message here and I’ll pinpoint the next step. Docs: - Event triggers: https://docs-3.prefect.io/v3/concepts/event-triggers - Create an automation to run a deployment: https://docs-3.prefect.io/v3/how-to-guides/automations/creating-deployment-triggers