Thanks for the detailed context — that’s frustrating. Let’s narrow it down and keep you protected in the meantime. What we know and what it likely isn’t - Prefect OSS 3.x does not have any background job that “replays” or re-creates old events on its own. New rows in the events table almost always mean something is POSTing to /events (either your own code, an automation action that emits events, or another integration). - Automations can fire on old events, but they do not insert new events. Since you’re seeing new rows appear, there is almost certainly an emitter posting them. A few plausible culprits to investigate - A producer process that buffers/retries events and flushes a backlog on restart or on network recovery (e.g., a service you run that uses the Events client with retries). - An automation that includes “Emit event” as an action and is creating follow-on events with your custom name (look for “EmitEventAction” in your automation actions). - A process posting events with fresh IDs but old occurred timestamps (so “old events” appear “new” to Prefect). - Less common: multiple copies of your event producer (e.g., two replicas in K8s) waking up after a crash/eviction and replaying queued data. Immediate guardrails to stop storm re-firing - Add an “occurred within” constraint to your automations so they ignore stale events. In Prefect 3.x, automations can filter by event time — set the trigger to only consider events whose occurred timestamp is within your desired window (e.g., last 30–60 minutes). This prevents old re-emitted events from firing 50 actions at once. - If you are intentionally emitting these events, include a unique, high-entropy event

id

of your own and log it in your system so you can correlate “who emitted what and when”. Quick data checks to diagnose the source Run these against your Prefect database (Postgres) around the time of the storm. Replace the event name as needed. 1) Confirm they’re truly new inserts and see how old they “occurred”

SELECT id, event, occurred, recorded
FROM events
WHERE event = 'myprefect.webhook.generic.actor-hook'
  AND recorded > now() - interval '1 day'
ORDER BY recorded DESC
LIMIT 200;

- If

recorded

is “now” but

occurred

is weeks ago, something is re-emitting old events now. 2) See if you’re getting fresh IDs vs duplicates

SELECT 
  count(*) AS total_rows,
  count(DISTINCT id) AS distinct_ids
FROM events
WHERE event = 'myprefect.webhook.generic.actor-hook'
  AND recorded > now() - interval '1 day';

- If distinct_ids == total_rows, they’re new event IDs (i.e., newly emitted). - If not, something odd is happening with duplicate inserts (less likely). 3) Look for any Prefect automation action that might emit this event - In the UI, check the automations that fire around that time for an action like “Emit event”. - If you manage automations as code, search for something like

EmitEventAction

or “emit event” and your custom event name. Trace the emitter - Check server access logs for POST /events at the time the rows appear. This is the most reliable way to identify the source (container/pod, IP, or API key). - If running via Docker/Compose/Helm behind a proxy, check your reverse proxy (nginx, ALB, etc.) logs for requests to /events. - If you can, enable request/access logging at the API layer around the event storm window and look for Authorization headers (which API key/workspace was used). - If you control the custom code that emits these events, add explicit logging of the event

id

,

occurred

, and a “source” tag in the event payload so you can query it later. A few questions to help pinpoint root cause - What Prefect version are you on (server and clients)? Have you upgraded recently? - How is Prefect deployed (Docker Compose, Helm/Kubernetes, bare Docker)? Any periodic restarts/rollouts around every 3–4 weeks? - Do you have any automations that include an “Emit event” action?

- When the storm hits, are the new rows using new event IDs, and what are the

occurred

vs

recorded

timestamps? - How does your custom emitter send events (which client/API, any retry/backoff/queue)? Any service that could be buffering and then flushing after downtime? Useful references - Events concepts: Events - Automations concepts: Automations - Server events code (for deeper inspection): prefect/server/events - Actions (includes EmitEventAction): server/events/actions - Clients: server/events/clients.py - Triggers services: server/events/services/triggers.py If you can share: - One example row (id, event, occurred, recorded) from a “reappeared” event - Whether those IDs are brand new or match historical IDs - Your deployment details and Prefect version …I can help you trace the likely emitter and suggest a precise fix. In parallel, I’d strongly recommend adding the “occurred within last X minutes” filter to your automations so this can’t blow up again.

That snippet is very telling:

occurred: 2025-12-04T10:45:15.333975Z
received: 2025-12-09T08:00:23.202746Z

Interpretation: - occurred = when the producer says the event happened - received = when Prefect’s API actually received the event So this event was newly posted to Prefect on Dec 9, but it describes something that happened on Dec 4. In other words, something emitted this event “late” (or replayed it) on Dec 9. That aligns with your automations firing in bulk: Prefect sees a new event at received time and evaluates automations, even if the occurred time is older. Immediate next steps to pinpoint the source - Check API/proxy logs for POST /events around 2025-12-09 080023Z to identify the emitter (container/pod/IP/API key). - Verify you do not have an automation action that emits your custom event name. Search your automations for an “Emit event” action using

myprefect.webhook.generic.actor-hook

. DB queries that can help 1) List “new” inserts for your custom event and compare occurred vs recorded

SELECT id, event, occurred, recorded
FROM events
WHERE event = 'myprefect.webhook.generic.actor-hook'
  AND recorded > now() - interval '1 day'
ORDER BY recorded DESC
LIMIT 200;

2) Confirm whether they’re new IDs (i.e., not duplicates)

SELECT 
  count(*) AS total_rows,
  count(DISTINCT id) AS distinct_ids
FROM events
WHERE event = 'myprefect.webhook.generic.actor-hook'
  AND recorded > now() - interval '1 day';

How to prevent future storms while you investigate - Add a time window to your automation trigger so it only reacts to events whose occurred is within your acceptable freshness, e.g., last 30–60 minutes. This ensures late/replayed events won’t trigger 50 actions at once. - If you control the emitter, make sure it does not buffer/retry indefinitely, and consider setting a max “age” for events it will send. - If your emitter can set the event

id

, you can use a stable idempotency key to avoid accidental re-sends of the same logical event. A couple of clarifying questions to help us zero in: - Do you have any automations with an “Emit event” action that might be producing

myprefect.webhook.generic.actor-hook

? - How is Prefect deployed (Compose/K8s/etc.) and do you see periodic restarts/rollouts around the times these storms happen? - What version of Prefect Server are you on? - Can you share one example row with

id, event, occurred, recorded

from one of these late events? References - Events: Events - Automations: Automations Based on the timestamps you shared, this was a new POST to /events on Dec 9 with an older occurred time. The quickest mitigation is tightening the “occurred within” window on your automations; then we can track down the emitter via logs.

Thanks — this confirms what’s happening. Your automation fired at 080036Z because a new event was received by the API at 080023Z: - triggering_event.occurred = 2025-12-03T104648.019Z (when the emitter says it happened) - triggering_event.received = 2025-12-09T080023.193Z (when Prefect actually received it) So this was not a historical event being “replayed” by Prefect — something posted a fresh event to /events on Dec 9 with an older occurred time. That’s why you see a burst of automations: Prefect has no way to know it’s “stale” unless you enforce a freshness window in the trigger. What to do next (concrete steps) 1) Trace the emitter at 080023Z - Check your reverse proxy/load balancer or API logs for POST /events at 2025-12-09 080023Z. Identify the client IP or API key that made the request. - If you’re on K8s with an ingress (e.g., NGINX), grep access logs for /events around that minute. - If you can’t see the origin easily, temporarily rotate the API key used by your suspected emitter; if the late posts stop, you’ve found the source. 2) Inspect the event row by id in your DB Use the triggering id from your payload (019ae3d2-65d3-7442-856d-c9a5844da0e8):

SELECT id, event, occurred, received, recorded, resource, related, payload
FROM events
WHERE id = '019ae3d2-65d3-7442-856d-c9a5844da0e8';

- recorded is when Prefect persisted the row. - If you see a “fresh” recorded/received with an older occurred, the emitter posted this late. 3) Add a freshness window to your automation trigger Configure your automation to only react to events whose occurred is within your acceptable window (e.g., last 30–60 minutes). This prevents late or replayed events from firing: - UI: Edit automation -> Trigger -> “Event occurred within last …” - Docs: Automations and Events 4) Audit your emitter(s) The event shows: - event: ifxprefect.webhook.generic.promos-cache - resource.id: ifxprefect.webhook, resource.name: Generic Webhook That strongly suggests a custom service you run is posting these. Common causes: - Buffered/retry logic that flushes a backlog on restart/network recovery - Multiple replicas emitting after a crash/eviction - A job re-sending older events without a max-age guard Recommended hardening - In your emitter, enforce a max age: do not send events older than X minutes. - Include a clear “source” in payload or as a related resource so you can trace origins quickly next time. - Optionally restrict POST /events to your trusted network or service account. - If you ever emit via an automation action, confirm you do not have an “Emit event” action producing these same custom events. If you can share: - Whether any POST /events hits were logged at 080023Z (IP/API key) - The DB row for that event id (occurred/received/recorded) - How your emitter is deployed (K8s/Compose) and if it restarts around every 3–4 weeks …I can help pinpoint the exact component and suggest a targeted fix. In parallel, the freshness filter on the automation will stop these storms immediately.

Short answer: there isn’t a direct “ignore events older than X minutes” field in Prefect’s EventTrigger. The

within

field is a counting window, not a freshness filter. To enforce recency at the automation level, the reliable pattern is to anchor your trigger with an “after” event that you emit regularly (e.g., a heartbeat), then bound the window with

within

. Late events that occurred before the last heartbeat will be ignored. Two JSON examples you can use: 1) Reactive EventTrigger: only react to your event if it occurs within 60 minutes after your latest heartbeat - Prereq: you emit

my.system.heartbeat

regularly (e.g., every minute)

{
  "name": "promos-cache (fresh within 60m)",
  "description": "Trigger only on promos-cache events that occur within 60 minutes after the latest heartbeat",
  "enabled": true,
  "trigger": {
    "type": "event",
    "posture": "Reactive",
    "after": [
      "my.system.heartbeat"
    ],
    "expect": [
      "ifxprefect.webhook.generic.promos-cache"
    ],
    "threshold": 1,
    "within": 3600,
    "for_each": [
      "prefect.resource.id"
    ],
    "match": {
      "prefect.resource.id": "ifxprefect.webhook"
    },
    "match_related": {}
  },
  "actions": [
    {
      "type": "do-nothing"
    }
  ]
}

How it works: - The “after” event sets the bucket start at the heartbeat’s occurred time. -

within: 3600

seconds means only events that occur within 60m after that heartbeat are counted. - Any late/replayed event whose occurred is before the last heartbeat will be ignored by the trigger logic. 2) SequenceTrigger alternative: require heartbeat then your event within 60 minutes - Same prereq: emit

my.system.heartbeat

regularly

{
  "name": "promos-cache (sequence within 60m)",
  "description": "Require a heartbeat then a promos-cache event within 60 minutes",
  "enabled": true,
  "trigger": {
    "type": "sequence",
    "within": 3600,
    "triggers": [
      {
        "type": "event",
        "posture": "Reactive",
        "expect": [
          "my.system.heartbeat"
        ],
        "threshold": 1,
        "within": 0
      },
      {
        "type": "event",
        "posture": "Reactive",
        "expect": [
          "ifxprefect.webhook.generic.promos-cache"
        ],
        "threshold": 1,
        "within": 0,
        "match": {
          "prefect.resource.id": "ifxprefect.webhook"
        }
      }
    ]
  },
  "actions": [
    {
      "type": "do-nothing"
    }
  ]
}

Notes and recommendations - There is no built-in “occurred in the last N minutes (relative to now)” field in EventTrigger JSON. The above patterns use “after” +

within

to bound acceptable occurred times. - If you control the emitter, also enforce freshness there: do not POST /events if

now - occurred > 60m

. - As a quick stop-gap, you can also narrow the automation’s scope with `match`/`match_related` to reduce blast radius. Docs - Automations: Automations - Events: Events If you share your current automation JSON, I can adapt it to add the heartbeat pattern with the correct fields for your setup.

Great question. In Prefect 3.x, you enforce a “freshness window” on a reactive EventTrigger with the

within

field (seconds). Here’s a minimal JSON you can POST to /api/automations that only fires if the event occurred within the last 60 minutes:

{
  "name": "Promos cache webhook - recent only",
  "description": "Fire only when promos-cache webhook events are fresh (<= 60 min old)",
  "enabled": true,
  "tags": [
    "webhook",
    "promos-cache"
  ],
  "trigger": {
    "type": "event",
    "posture": "Reactive",
    "expect": [
      "ifxprefect.webhook.generic.promos-cache"
    ],
    "threshold": 1,
    "within": 3600
  },
  "actions": [
    {
      "type": "send-notification",
      "block_document_id": "YOUR-NOTIFICATION-BLOCK-ID",
      "subject": "Promos-cache webhook received",
      "body": "Event {{ trigger.triggering_event.id }} occurred at {{ trigger.triggering_event.occurred }}"
    }
  ]
}

- Set

within

to your freshness window in seconds (e.g., 1800 for 30 minutes). -

posture: "Reactive"

+

threshold: 1

means “fire when at least one matching event occurs within the window.” If you also want to scope to a specific resource, you can add a

match

filter (optional):

"match": {
  "prefect.resource.id": "ifxprefect.webhook"
}

Docs for context: - Automations: Automations - Events: Events If you’d like, share your current automation JSON and I’ll adjust it to add the freshness window precisely.