Short answer: this is usually caused by UTC boundary effects and timing windows in Prefect Cloud 2.x Automations. Around midnight, a few things can combine to make notifications appear “missed”: - Timezone/UTC mismatch: Automations evaluate times in UTC. If your logic assumes local midnight, your trigger windows may be misaligned at 00:00 UTC. - Bucket boundary at midnight: the event aggregator rolls buckets by time window; events arriving right at a boundary can fall outside the active bucket. - Lookback/dedup windows: there’s a 15-minute preceding-event lookback and a 30-minute action de-dup window; high load or slight delays near midnight can cause triggers or actions to be skipped as “already processed.” - Proactive triggers timing: proactive automations re-evaluate every few seconds; if many triggers expire at midnight, you can get thundering-herd timing or races. - Provider reliability: Slack/email/webhook providers often rate-limit or have maintenance at :00 UTC; the action may “succeed” or “fail” depending on the provider response at that moment. Helpful references: - Automations (Prefect 2.x) - Events in Cloud For context, some relevant constants in 2.x:

# Event processing windows (Prefect 2.x)
PRECEDING_EVENT_LOOKBACK = 15 minutes
PROCESSED_EVENT_LOOKBACK = 30 minutes
# Proactive evaluation granularity defaults to ~5 seconds

To pinpoint your case, could you share: - Midnight in which timezone? - The automation definition (trigger type, posture reactive vs. proactive, any “within” window, and the action used)? - Provider used for notifications (Slack, email, Teams, webhook)? - In the automation’s Activity, do you see the trigger firing but the action failing, or no trigger at all? Quick checks you can run now: - In the Automation detail page > Activity: confirm if it triggered at that time and inspect the triggered action details for any notification errors. - Cloud Events feed: filter for your failure event (e.g.,

prefect.flow-run.failed

) around midnight UTC to confirm the event landed when you expect. - If using a proactive “within X” window that starts/ends at midnight, shift those boundaries away from 00:00 UTC and retest. - Test your notification block directly (send a test message) around that time to rule out provider rate limits. Mitigations that typically resolve midnight gaps: - Avoid windows that start/end at 00:00 UTC; use 00:05 or another off-peak boundary. - Prefer reactive triggers for flow-run failures instead of proactive windows when possible. - Add a backup action (e.g., webhook) so a provider hiccup doesn’t drop the alert. - If DST is in play, confirm you’re not depending on local midnight semantics—Automations run in UTC. If you can share the automation’s JSON or a screenshot of the trigger/action config, I can help spot the exact issue and propose a precise fix.

Thanks for the details — this helps a lot. What’s happening - Your automation only fires when the event has the related resource prefect.tag.prod:

"match_related": {
    "prefect.resource.id": ["prefect.tag.prod"],
    "prefect.resource.role": "tag"
  }

- In Prefect 2.x, a child (subflow) flow-run event does not automatically inherit the parent’s tags. The event’s related tags are built from the child flow/deployment/work pool/queue, not the parent. So if the child flow/deployment doesn’t have the prod tag, its Failed event will be filtered out by your automation and no Slack message is sent. - Your note that “parent flow and child flow share the same timestamp” lines up with this: overnight, it’s likely the child fails (no prod tag) and the parent either doesn’t fail or fails slightly later, so the automation misses some of those child failures. - The “works after 10am” pattern likely correlates with different runs (e.g., top-level parent failures or runs whose child flows/deployments do carry the prod tag), not a UTC/midnight issue. Brisbane is UTC+10, so 000–900 AEST is 1400–2300 UTC — not a known boundary for Prefect’s event windows. How to confirm - Open one missed event in Cloud > Events and check Related resources. You’ll see the child flow-run event lacks prefect.tag.prod. - Check the parent’s flow-run.Failed event (if it exists) around the same time — it should have the tag and would match your automation. - Docs for how Automations match on related resources and tags: Automations (2.x), Cloud Events Fix options Pick one of these based on how you separate prod/non-prod: 1) Propagate prod to subflows (recommended) - Ensure the child flow runs in prod carry the prod tag. - Easiest approach: call the subflow with tags from the parent in prod.

from prefect import flow

  @flow(name="child-flow")
  def child(...):
      ...

  @flow(name="parent-flow", tags=["prod"])
  def parent(...):
      # In prod, ensure the child subflow run is tagged 'prod'
      child.with_options(tags=["prod"])(...)

- Or add the prod tag to the child deployment(s) used in prod so all child runs in prod inherit it. 2) Match on deployment/flow/work-pool instead of a tag - If you have prod-specific deployments or a prod work pool, filter on those instead of tags. For example:

"match_related": {
    "prefect.resource.role": "work-pool",
    "prefect.resource.name": ["prod-pool"]
  }

or

"match_related": {
    "prefect.resource.role": "deployment",
    "prefect.resource.name": ["my-prod-deployment-*"]
  }

- This avoids tag propagation entirely and keys the alerting to your prod infra objects. 3) Keep current automation for parent failures and add a second automation for child failures - A second automation without the prod tag filter (or filtered by child’s prod deployment) ensures subflow failures are caught as well. Why this isn’t a UTC/midnight thing - Your window (000–900 AEST) doesn’t align with any Prefect timing boundaries (all internal evaluation is UTC). - Slack webhook rate limiting could explain sporadic misses but not a consistent 9-hour window. The tag filter mismatch is the consistent explanation for “child fails, no alert.” If you share one missed event’s Related list (screenshot or JSON), I can confirm the absence of prefect.tag.prod and propose the exact match changes for your automation.

Great detail — thank you. A few key points and a plan to isolate the cause: Direct answers to your questions - Same timestamp: does not suppress triggers. Reactive automations fire per event; timestamps aren’t used for dedup. - Same flow-run ID: parent and child never share the same flow-run ID. They’re distinct resources, so your

for_each: ["prefect.resource.id"]

won’t suppress one because of the other. Why your added automations could still be silent at the same times - Flow ID pinning: Your flow-scoped automation matches a single flow ID:

"match_related": {
    "prefect.resource.role": "flow",
    "prefect.resource.id": ["prefect.flow.87cb53ea-6126-4439-9e82-b424ef7fe0f9"]
  }

If the flow was re-registered or runs under a different flow object (same name, new ID), events will not match. This can happen if deployments or environments recreate the flow object. The “worked in daytime” and “silent at night” could indicate different process paths creating runs with a different flow id at night. - Work pool filter: The flow-run event must include the work-pool as a related resource for the pool automation to match. Subflows sometimes do not surface the pool relationship the way top-level runs do, and ad-hoc or locally-triggered runs may lack a pool relationship. - Schedule filter: This one is likely a no-op. Flow-run events generally do not include a related

prefect.schedule.*

resource, so it won’t match those events. - Provider side: If all automations “triggered” but you saw no Slack posts, a webhook-level issue could explain synchronized silence (429 rate limits, transient provider issues). If no automations triggered at all in Activity, it’s a matching issue rather than Slack. What to check next (will pinpoint root cause quickly) 1) Pick one missed parent flow-run.Failed event and open it in Cloud > Events. In Related resources, confirm: - The exact flow resource present (id and name) - Whether a work-pool appears (and which one) - Any tags present 2) Open the Automation’s Activity for the same time: - If there’s no activity: it didn’t match; adjust filters as below. - If there is activity and an action record: open the action details — look for a 429 or 5xx from Slack/webhook. 3) If possible, share the event JSON’s related resource list here; I can map it precisely to your automation filters. Recommended fixes - Avoid pinning to a specific flow ID. Use flow name or deployment instead:

"match_related": {
    "prefect.resource.role": "flow",
    "prefect.resource.name": ["<parent flow name>"]
  }

or match by deployment:

"match_related": {
    "prefect.resource.role": "deployment",
    "prefect.resource.name": ["<prod deployment name>"]
  }

This prevents “silent” periods if the flow ID changes overnight. - For pool scoping: confirm missed events actually include a work-pool related resource with one of those IDs. If they don’t, match on deployment or flow instead. - Remove the schedule-based automation (it won’t match flow-run events). If you want scheduled-run failures specifically, target deployments or flows, not

prefect.schedule.*

. - Add a “shadow” action for diagnostics: - Create a temporary automation that matches all flow-run failures and posts to a RequestBin/Hookbin URL. If you see those HTTP hits during the silent window, but Slack is quiet, it’s the Slack webhook pathway. If you don’t see hits, matching is the culprit. - If you want child failures covered, either: - Propagate prod tag to subflows (e.g.,

child.with_options(tags=["prod"])

), or - Create a child-specific automation matched by the child’s flow or deployment (by name, not id). Why this explains “all silent at the same time”

- If the nighttime runs are created by a slightly different registration/deployment path (new flow id, different pool/queue), your automations pinned to specific related resources won’t match. All those automations would be silent together because they all depend on related-resource matches that are absent or different during that window. - Alternatively, if automations show as triggered in Activity but no Slack posts arrive, that implicates the shared Slack webhook or workspace policy (rate limiting or network policy) — all automations share the same outbound path so they’d be “silent” together. References - Automations and related resource matching: https://docs.prefect.io/2.14/concepts/automations/ - Events model and related resources: https://docs.prefect.io/2.14/cloud/events/ If you can paste the Related section from one missed parent flow-run.Failed event, I’ll suggest the exact

match_related

you should use.

Thank you for the JSON — that pinpoints it. What the event shows - The automation did trigger, but the action failed. - Action type: call-webhook (using your block martech-slack-webhook). - Slack responded 400 invalid_payload. - Headers (x-slack-*) confirm this is Slack’s Incoming Webhooks endpoint. Conclusion - This is not a timing/UTC or dedup issue. Slack is rejecting the JSON body Prefect is sending at that time with “invalid_payload”. All automations that use this same webhook/payload will “go silent” together because they all hit the same Slack rejection. Why it may happen “only at night” - Your payload template likely references event fields that are sometimes missing (e.g., deployment/work-pool/parent info). When those fields are absent (common with subflows or certain run paths, which for you happen more often between 000–900 AEST), the rendered JSON ends up violating Slack’s schema: - Missing required top-level text when blocks are present, or - Empty blocks array, or - A block with a missing nested text field, or - A field rendered as null/empty where Slack expects a non-empty string/array of valid objects. - Slack returns 400 invalid_payload for schema violations (different from 429 rate limits or invalid_json). What to do next 1) Inspect your Webhook block payload/template - In the “martech-slack-webhook” block (or the Automation action override), check the body you send to Slack. - Make sure it always includes either: - a non-empty "text" string; or - a valid "blocks" array with at least one proper block object (and text fields present where required). - Avoid empty arrays and nulls. Use defaults and conditionals. 2) Make the payload schema robust to missing fields - Use defaults to avoid nulls: - Jinja examples:

{{ value | default('N/A') }}

and

{{ long_text | truncate(2800) }}

- Only include a block if content exists: -

{% if some_value %}{ ... }{% endif %}

- Don’t send empty "blocks": [] or a section block without a valid text object. 3) Minimal working payload to test - Temporarily set the Webhook body to this known-good shape and test around the failing window:

{
  "text": "*Prefect alert:* A flow run failed.",
  "mrkdwn": true
}

- If this succeeds at midnight, the issue is definitely in your templating logic for richer messages. 4) Example robust template (with conditionals)

{% set flow = (event.related | selectattr('prefect.resource.role','equalto','flow') | list | first) %}
   {
     "text": "*Prefect alert:* Flow run failed",
     "blocks": [
       {"type":"section","text":{"type":"mrkdwn","text":"*Flow:* {{ flow['prefect.resource.name'] | default('unknown') }}"}},
       {"type":"section","text":{"type":"mrkdwn","text":"*Run:* {{ event.resource['prefect.resource.id'] }}"}},
       {% if event.payload and event.payload.get('state', {}).get('message') %}
       {"type":"section","text":{"type":"mrkdwn","text":"```{{ event.payload.state.message | truncate(2800) }}```"}}
       {% endif %}
     ]
   }

5) Consider using “Send a notification” with a Slack Webhook notification block - It handles the Slack payload shape for you. In Automations, choose “Send a notification” and select your Slack Webhook block. This is generally safer than a custom Webhook payload. 6) Add a temporary “debug sink” action - Add a second action that POSTs the same payload to a RequestBin/Hookbin URL so you can inspect the exact JSON produced during the failing window. References - Prefect Automations overview: https://docs.prefect.io/2.14/concepts/automations/ - Slack Incoming Webhooks schema: Slack incoming webhooks docs If you can share the current payload template from martech-slack-webhook (or a screenshot), I’ll point out exactly where it can produce an invalid Slack payload and propose a corrected version.

Thanks for sharing the payload. The automation event you posted shows Slack is returning 400 invalid_payload, so the issue is with the body Slack receives. Your template is close, but a few fields can make the payload invalid when they’re missing/huge/unexpected at night. Potential culprits in your current template - flow_run.state.message - Can be None or extremely long (stack traces). Very long messages can trigger invalid_payload. Also, if the message contains unescaped characters, it can break JSON unless you escape it. - flow_run.state.timestamp - Can be absent early in failure transitions. Jinja will render “None” as a string, which is okay, but if it’s a non-string object in some contexts it can render oddly. - flow.name or flow_run.name - Usually present, but if the context differs (e.g., certain subflow or system transitions), these could render empty. - ui_url filter - If the filter is not available in this templating context, it would normally raise before the webhook call. Since Slack returns 400, Jinja did render something. Still, the value could be empty depending on context. Additionally: - Very long text: Slack rejects messages that exceed its limits (text length/overall payload). This often varies with error messages — if your failures at night produce larger exception messages, Slack will reject them while daytime messages pass. How to make this payload robust - Always provide defaults, escape, and truncate long fields (especially state.message). - Let Jinja produce a valid JSON string for text using tojson (escapes quotes/newlines safely). - Consider removing channel/username/icon_emoji from the payload; for many newer incoming webhooks these are ignored or restricted. If you keep them, ensure the webhook is allowed to post to that channel. A safer version of your payload Replace your payload with this. It’s functionally the same but robust to missing/huge values and ensures valid JSON.

{
  "text": {{ (
    "Martech-Slack-Error-Alerts-ML-Test-2:\n\n" +
    "```" +
    "Flow run: " + (flow.name | default("unknown")) + "/" + (flow_run.name | default("unknown")) + "\n" +
    "State: `" + (flow_run.state.name | default("unknown")) + "`\n" +
    "Timestamp: " + ((flow_run.state.timestamp | default(flow_run.start_time) | string) | default("unknown")) + "\n" +
    "Flow run URL: " + ((flow_run | ui_url) | default("")) + "\n" +
    "State message: " + ((flow_run.state.message | default("") | truncate(2800))) +
    "```"
  ) | tojson }}
}

Notes: - tojson ensures the text value remains valid JSON regardless of embedded quotes or newlines. - truncate(2800) keeps the code block well within Slack limits. - default(...) guards against None/missing values. If you prefer to keep channel/username/icon_emoji, you can add them back; just be aware some Slack webhooks ignore or reject overrides depending on your app configuration. Quick validation steps - Temporarily switch to a minimal payload and test during the problem window:

{
  "text": "Prefect alert: a flow run failed."
}

If this succeeds, the previous payload was invalid due to content. - Add a second “debug” webhook action to a RequestBin/Hookbin URL to capture the exact JSON Prefect sends when Slack returns 400 — you’ll quickly see if state.message is huge or if any field is blank/unexpected. - Consider using “Send a notification” with a Slack Webhook notification block in the Automation; Prefect will format a safe Slack payload for you. Reference - Slack incoming webhook payload rules: https://api.slack.com/messaging/webhooks If you paste one failing RequestBin capture (redacted), I can point out the exact field causing Slack’s 400.

Short answer: the risky field is the state message. When you inline values like flow_run.state.message directly into a JSON string without escaping, any embedded double quotes, backslashes, or unusual characters will break the JSON you send to Slack and cause 400 invalid_payload. That’s why your minimal payload with just flow.name works and the richer one sometimes fails. What typically flips “only at night” - flow_run.state.message often contains exception text with quotes, backslashes, or very long content. Some of your night failures likely include characters that must be escaped in JSON, so Slack rejects the payload. - Less likely but possible: flow_run.state.timestamp is None, or other fields contain unexpected content; these won’t usually cause a 400 unless they introduce unescaped quotes. The state message is the prime suspect. How to fix reliably - JSON-encode the entire text value using tojson so quotes/newlines are escaped. - Add defaults and truncate the message to keep it reasonable. Try this safer payload:

{
  "channel": "ml-test-alerts",
  "username": "Prefect-Alert-Bot",
  "icon_emoji": ":prefect:",
  "text": {{ (
    "Martech-Slack-Error-Alerts-ML-Test-2:\n\n" +
    "```" +
    "Flow run: " + (flow.name | default("unknown")) + "/" + (flow_run.name | default("unknown")) + "\n" +
    "State: `" + (flow_run.state.name | default("unknown")) + "`\n" +
    "Timestamp: " + ((flow_run.state.timestamp | default(flow_run.start_time) | string) | default("unknown")) + "\n" +
    "Flow run URL: " + ((flow_run | ui_url) | default("")) + "\n" +
    "State message: " + ((flow_run.state.message | default("") | truncate(2800))) +
    "```"
  ) | tojson }}
}

Why this works - tojson escapes quotes, backslashes, and newlines so Slack gets valid JSON every time. - default(...) prevents None from rendering as bare nulls. - truncate(2800) keeps giant stack traces from bloating the message. Optional checks - If you still see 400s, try a minimal payload again:

{
  "text": "Prefect alert: a flow run failed."
}

If that works while the rich one fails, the issue is definitely content/escaping. - You can also POST the rendered payload to a RequestBin/Hookbin to inspect exactly what Slack is receiving. References - Slack incoming webhooks - Prefect 2.x Automations

Good catch — the template you pasted has two problems that will cause invalid_payload even with tojson: 1) Jinja concatenation: use ~ instead of + 2) Braces: you have an extra { after "text": and too many closing braces at the end Here’s a corrected, robust version that should work. It JSON-encodes the whole text field, handles missing values, and truncates long messages.

{
  "channel": "ml-test-alerts",
  "username": "Prefect-Alert-Bot",
  "icon_emoji": ":prefect:",
  "text": {{ [
    "Martech-Slack-Error-Alerts-ML-Test-2:",
    "Flow run: " ~ (flow.name | default("unknown")) ~ "/" ~ (flow_run.name | default("unknown")),
    "State: " ~ (flow_run.state.name | default("unknown")),
    "Timestamp: " ~ ((flow_run.state.timestamp | default(flow_run.start_time) | string) | default("unknown")),
    "Flow run URL: " ~ ((flow_run | ui_url) | default("")),
    "State message: " ~ ((flow_run.state.message | default("") | truncate(2800)))
  ] | join("\n") | tojson }}
}

Why this should fix it - Single set of {{ }} around the expression after "text": … - ~ is the Jinja string concatenation operator - tojson escapes quotes, backslashes, and newlines so Slack gets valid JSON - default(...) prevents None - truncate avoids oversized payloads If you prefer the previous “single string” style, this is also valid:

{
  "channel": "ml-test-alerts",
  "username": "Prefect-Alert-Bot",
  "icon_emoji": ":prefect:",
  "text": {{ (
    "Martech-Slack-Error-Alerts-ML-Test-2:\n" ~
    "Flow run: " ~ (flow.name | default("unknown")) ~ "/" ~ (flow_run.name | default("unknown")) ~ "\n" ~
    "State: " ~ (flow_run.state.name | default("unknown")) ~ "\n" ~
    "Timestamp: " ~ ((flow_run.state.timestamp | default(flow_run.start_time) | string) | default("unknown")) ~ "\n" ~
    "Flow run URL: " ~ ((flow_run | ui_url) | default("")) ~ "\n" ~
    "State message: " ~ ((flow_run.state.message | default("") | truncate(2800)))
  ) | tojson }}
}

Optional tips - If you still see 400, temporarily remove the channel/username/icon_emoji fields and send only the text field; some Slack webhooks disallow channel overrides depending on the app’s configuration. - To verify exactly what’s being sent, add a second action to POST to a RequestBin/Hookbin URL and inspect the rendered JSON. - Alternatively, switch the action to “Send a notification” with a Slack Webhook notification block; Prefect will handle Slack-safe formatting for you.