<@ULVA73B9P> Using a Prefect 3 self hosted server,...
# ask-marvin
n
@Marvin Using a Prefect 3 self hosted server, we are using a flow deployed such that it triggers on events. These events happen very often, and the triggers sometimes are registered minutes after the actual event occurred. Why?
m
thought for 250.9 seconds
Short answer: in Prefect 3 the event→trigger path is intentionally batched/polled. Under high event rates or a slow database, the batching queues can back up, so a trigger may be recorded well after the event’s timestamp. The most common contributors are below plus how to tune them. Why this happens - Event persister batching (biggest factor by default) - The server’s EventPersister buffers events and only flushes to the DB when either the batch fills or a flush interval elapses. Defaults are small batches and a 5s flush window. - Under high volume or slow DB (especially SQLite), the in‑memory queue can backlog and add seconds→minutes of delay before your trigger service can even see the event. - Proactive trigger polling cadence - If your automation is “absence”-based (proactive), it’s evaluated on a fixed cadence (default ~5s). It won’t fire until the next tick. - Causal ordering waits - If an event includes a follows dependency and its “leader” hasn’t been processed yet, followers can be held briefly; in edge cases where the leader never arrives, a follower can be delayed significantly before being released. - Database slowness/connection limits - Remote DB latency, small connection pools, or SQLite single-writer locks slow down event flushes and trigger bucket reads/updates, causing backlogs. - Sheer volume - If you’re emitting more events/sec than the server can persist, backlog accumulates and you’ll see “minutes late” until the pipeline catches up. What to tune (biggest impact first) - Make the persister flush more frequently and in bigger chunks - Reduces worst-case “wait to flush” and improves throughput. - Set and restart the server services:
Copy code
export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_FLUSH_INTERVAL=1      # default 5
  export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_BATCH_SIZE=10         # default 20
  export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_READ_BATCH_SIZE=20    # default 1
  export PREFECT_SERVER_SERVICES_EVENT_PERSISTER_QUEUE_MAX_SIZE=100000 # default 50000
- Speed up trigger consumption
Copy code
export PREFECT_SERVER_SERVICES_TRIGGERS_READ_BATCH_SIZE=20           # default 1
- If using proactive/absence triggers, poll more often
Copy code
export PREFECT_EVENTS_PROACTIVE_GRANULARITY=1                        # default ~5
- Use PostgreSQL instead of SQLite - SQLite is single-writer and will bottleneck quickly under frequent events. Point Prefect to Postgres for production. - Keep the DB close and healthy - Minimize network latency, ensure sufficient connections, and consider lowering event retention if your event tables are huge. What to check - Server logs for backlog clues: - Messages about “Event queue … capacity” or repeated flush retries indicate the persister is backlogged or the DB is slow. - Your DB backend and size: - Are you on SQLite? How big is the events table? Any connection pool exhaustion? - Trigger posture: - Is your automation reactive (on event) or proactive (absence/polling)? - Event rate: - Roughly how many events/second at peak? Useful docs - Self-hosted scaling and performance: Self-hosted (Prefect 3) - Event triggers concept: Event triggers - Creating deployment triggers: Creating deployment triggers If you share a bit more about your setup (DB type, approximate event rate, whether your triggers are reactive vs proactive, and any log snippets), I can suggest a concrete config that will bring your latency down from minutes to roughly 1–2s.