Short answer: there are a few stages between “emit” and “I see it/it fires a trigger” in Prefect 3.x. Delays usually come from network/API ingestion, server batching/ordering, trigger evaluation mode (reactive vs proactive), UI polling/querying, and system load/backpressure. Common causes of noticeable lag - Network/API ingestion - Emitting to Prefect Cloud adds network latency and can hit transient rate limits (429), which the SDK retries. This can add seconds under bursty load. - Event ingestion and batching - The server persists events in small batches for efficiency. Under load, or with larger batches, this adds a short delay before the event is committed. - Causal ordering/waiting for “preceding” events - If you send events with a `follows`/causal relationship, the server may briefly hold an event while it looks for/expects the earlier event. If the “preceding” event is late or missing, this can add delay. If you don’t need causal linking, omit it. - Trigger evaluation mode - Reactive triggers (when X happens) are near real-time once the event is ingested. - Proactive triggers (when X doesn’t happen within N) are evaluated on a periodic schedule; they won’t fire faster than that cadence and the window you set. Expect 10s+ resolution for absence checks. - UI freshness and query performance - The UI polls and queries the events table. On large data sets or under load, queries can take a few seconds. So you might see the trigger fire before the UI shows the event, or vice versa. - Backpressure / heavy load - High event volume, database contention, or message-queue backlogs can slow ingestion and trigger pickup. In Cloud, transient platform load can also contribute. What’s “normal” - Reactive trigger pickup: typically sub‑second to ~1s after ingestion in healthy conditions. - UI visibility: usually within a few seconds, but can stretch under load or with complex filters. - Proactive triggers: determined by your “within” window and the service’s sweep cadence; don’t expect sub‑second behavior. Quick checks and mitigations - Verify trigger type and window - If you need immediate reaction, use a reactive trigger. For absence-based logic, make the “within” window realistic (10s+). - Double-check the trigger filter - Ensure event name, resource/namespace attributes, and payload fields exactly match what your trigger subscribes to. - Confirm causal links aren’t blocking - If you’re setting

follows

but the “preceding” event is missing or delayed, try omitting

follows

to see if latency drops. - Watch for retries/rate limits - Look for 429 or retry logs where you emit events. Bursts can be smoothed (e.g., add small client-side backoff). - For self-hosted (OSS) - Ensure your server/postgres are well resourced. Large, unpartitioned event tables slow queries; consider retention tuning and partitioning. - Check server logs for the events persister/triggers services for backlogs or slow queries. If you can share a bit more, I can be more specific: - Are you on Prefect Cloud or self‑hosted? - Roughly how long is the lag (seconds/minutes)? Is it consistent or sporadic? - Is the trigger reactive or proactive? - Approximate event volume when you see the lag? - An example event name and when you sent it (timestamp), plus what the trigger filter looks like. Relevant docs - Event triggers overview: Event triggers in Prefect 3.x - Emitting events (SDK): Events utilities API - Events REST endpoint: Create events API - Triggers services (reactive/proactive): Triggers services - Database considerations: Database maintenance and performance

Thanks for the details — that delay is definitely higher than expected. Let’s isolate where the latency is coming from and apply a few server-side tuning options. I’ll assume Prefect 3.x self-hosted; if you’re on 2.x or a different setup, let me know. Where the delay can occur (event → run) - Event ingestion: events hit the API and get batched/persisted to Postgres - Automations: your event-based automation evaluates and fires an action (e.g., create flow run) - Run scheduling: the Scheduler loop materializes scheduled/pending runs - Worker pickup: workers poll for pending runs and start them Quick triage: pinpoint the bottleneck 1) Confirm events are being ingested - API: count recent events

curl -s -X POST "$PREFECT_API_URL/api/events/count-by/type" \
  -H "Authorization: Bearer $PREFECT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"filter":{"occurred":{"since":"5m"}}}'

- Stream: connect a WebSocket to your server’s events-out to watch live - ws URL is typically at /api/events/out (filterable) - DB (safest): on your Prefect Postgres

-- Do you see events arriving right away?
SELECT COUNT(*) 
FROM event 
WHERE occurred > NOW() - INTERVAL '5 minutes';

2) Check that an automation fires on those events - In the UI, open Automations and check “Last triggered” and recent activity. - DB (optional): do you see new flow runs right after relevant events?

SELECT id, created, state_type, deployment_id
FROM flow_run
WHERE created > NOW() - INTERVAL '15 minutes'
ORDER BY created DESC
LIMIT 50;

3) Check run scheduling latency - Are runs stuck in SCHEDULED for a long time before going PENDING?

SELECT id, created, expected_start_time, start_time, state_type
FROM flow_run
WHERE created > NOW() - INTERVAL '30 minutes'
ORDER BY created DESC
LIMIT 100;

- If they linger in SCHEDULED, the Scheduler loop is your likely bottleneck. 4) Check worker pickup - Worker logs: frequent polls and immediate submission after runs become PENDING - Ensure enough workers are online, and that work pool concurrency isn’t limiting. Server-side tuning for higher event volume If you’re doing a few thousand events/minute, these settings typically cut end-to-end latency substantially. Set them on your API server(s) and restart. - Event persister (batching/flush)

PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=500
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=2

- Scheduler (loop frequency and batch inserts)

PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS=30
PREFECT_SERVER_SERVICES_SCHEDULER_INSERT_BATCH_SIZE=2000

- Automations (time-based trigger evaluation granularity)

PREFECT_SERVER_EVENTS_PROACTIVE_GRANULARITY=2

- Messaging broker for events (recommended at your scale) - Default is in-memory; switch to Redis to decouple ingestion under load. Install the Redis integration and point the broker at it:

# Install integration in your server image/env:
# pip install "prefect-redis"  (or add to your image)

# Use Redis-backed messaging
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging

# Redis connection settings
PREFECT_REDIS_MESSAGING_HOST=redis
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# PREFECT_REDIS_MESSAGING_PASSWORD=...

Additional checks - Postgres health: watch CPU, IOPS, and connection limits. At high write rates from events + scheduler, under-provisioned Postgres will cause backpressure. - Indexes: ensure standard indexes exist; if your DB was migrated from an older version, missing indexes can slow queries. If you see slow queries in logs, we can dig into specific indexes. - Time sync: ensure all components (API, DB, workers) have synchronized clocks (NTP). Clock skew can manifest as “scheduled in the future” delays. - Worker capacity: add workers or increase concurrency for the affected work pool if runs transition to PENDING quickly but sit waiting for pickup.

If you can share a bit more, I can tailor the next steps: - Prefect version (e.g., 3.1.x) - How you’re hosting (Docker Compose, Helm/Kubernetes, other) - Postgres size and resources - Whether you’re currently using Redis for events messaging - Rough number of automations and the specific trigger types you rely on - A short snippet from your event persister, automations, and scheduler logs during a delay window - Worker type(s), count, and whether a work pool concurrency limit is set Helpful docs - Prefect Self-Hosting overview: Self-hosting Prefect - Events and Automations concepts: Automations and Events - Workers and Work Pools: Work Pools and Workers If you prefer, I can provide a step-by-step test plan with exact curl/SQL commands against your environment to identify whether the delay is at ingestion, automation firing, scheduling, or pickup.

Got it — if your DB shows 0 events while your app logs say “emitted”, the events likely aren’t reaching/persisting in the API. Let’s quickly isolate why and fix it. Most common causes - Multiple API replicas or multi-process server with the default in-memory event broker. Each process keeps its own in-memory queue, so the persister may never see events from another process. Fix: use Redis broker. - Event persister service disabled/crashing. Fix: enable and check logs. - Client posting to a different API URL/workspace. Fix: verify `PREFECT_API_URL`/`PREFECT_API_KEY` on the emitter. - HTTP posts failing silently (best-effort emitter). Fix: verify responses. Quick tests to confirm ingestion 1) POST a test event directly to your API and check the response

curl -i -X POST "$PREFECT_API_URL/api/events" \
  -H "Authorization: Bearer $PREFECT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "events": [
      {
        "occurred": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'",
        "event": "test.event.ingestion",
        "resource": {"prefect.resource.id": "demo:ingestion"},
        "payload": {"foo": "bar"}
      }
    ]
  }'

- Expect a 204. If not 204, share the status/body. 2) Watch events live via WebSocket (server-side) - Connect a WS client to:

ws://<your-api>/api/events/out

- If your test event doesn’t show up here, the API isn’t accepting/processing it. 3) Re-check DB directly after the test

SELECT COUNT(*)
FROM event
WHERE occurred > NOW() - INTERVAL '5 minutes';

If still 0, check these right away - Are you running multiple API replicas or a multi-process server? - If yes, switch to Redis for event messaging (in-memory broker is single-process only). Enable verbose event logging temporarily - This logs all received events at the API:

PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true

Restart the API and watch logs as you POST the test event. Recommended fix for multi-replica/multi-process - Install Redis messaging integration in your API image/env:

pip install "prefect-redis"

- Configure the broker to use Redis and restart API:

PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
PREFECT_REDIS_MESSAGING_HOST=redis          # or your host
PREFECT_REDIS_MESSAGING_PORT=6379
PREFECT_REDIS_MESSAGING_DB=0
# PREFECT_REDIS_MESSAGING_PASSWORD=...

Other things to double-check - Client config: - The process emitting events must have the same

PREFECT_API_URL

and an API key for the correct workspace. - If you’re using a Prefect client emitter, check logs for HTTP failures; the emitter may log “emitted” on enqueue, not on successful persistence. - Event persister status: - Ensure the API includes services; look for “event_persister” logs or errors on startup. - Table exists:

SELECT table_name
FROM information_schema.tables
WHERE table_schema='public' AND table_name LIKE 'event%';

If you can share: - How you’re running Prefect (Docker Compose vs K8s/Helm), and number of API replicas/processes - Values of

PREFECT_API_URL

, whether you use

PREFECT_API_KEY

- Whether you can reproduce with the curl test and the HTTP status code you get - A short snippet of your API logs around “event_persister” during the test Once we confirm ingestion works (DB count increases or WS shows events), we can tune batching/flush intervals and scheduler settings to reduce end-to-end latency. For reference: - Self-hosting overview: Self-hosting Prefect - Automations and events concepts: Automations and Events