Thanks for the report — that’s frustrating. A 10–15 minute lag on self-hosted automations typically comes from one (or a mix) of: - The background services that evaluate automations running too infrequently or being backlogged - Database contention/performance (especially if using SQLite) - Clock skew between containers/hosts - Using “proactive” triggers (absence-of-event) which are evaluated on a periodic loop A quick way to narrow it down 1) Confirm your setup - Prefect version (assuming 3.x): run

prefect --version

- Database: are you on PostgreSQL or SQLite? For production and lots of events/automations, PostgreSQL is required — SQLite will cause locking delays. - Deployment: Docker Compose vs Kubernetes, and how many “services” instances are running (generally only one services instance should run the triggers). 2) Make sure the services that power automations are running - In self-hosted, automations are evaluated by the services process (ReactiveTriggers, ProactiveTriggers, Actions, EventPersister). - If you use Docker Compose, there should be a “prefect-services” container. On K8s, the “services” deployment/pod needs to be up and healthy. 3) Check config (timing/loops) - View effective settings:

prefect config view --show-defaults

Useful keys to look for/tune: - Triggers service -

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED

(should be true) -

PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_HEARTBEAT_INTERVAL_SECONDS

(default ~5) -

PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_RECONNECT_INTERVAL_SECONDS

(default ~10) - Foreman (can add seconds to latency on state-driven automations) -

PREFECT_SERVER_SERVICES_FOREMAN_LOOP_SECONDS

(default ~15; consider 5–10) - Scheduler (doesn’t directly control automations but worth checking) -

PREFECT_SERVER_SERVICES_SCHEDULER_RECENT_DEPLOYMENTS_LOOP_SECONDS

(default ~5) - DB timeout (timeouts cause retries/backoff) -

PREFECT_SERVER_DATABASE_TIMEOUT

(consider 30–60s if your DB is under load) - Timezone/clock consistency - Set

TZ=UTC

everywhere; ensure NTP is syncing on all nodes Docs for settings and automations: - Settings reference - Automations and event triggers - Self-hosted (Docker Compose) - Helm (Kubernetes) 4) Look for proactive vs reactive automations - Reactive triggers fire near real-time on events (via Postgres LISTEN/NOTIFY). - Proactive triggers (e.g., “if X does NOT happen within N minutes”) are evaluated periodically; their effective “within” plus service loop adds inherent delay. If your automations look for the absence of an event, some delay is expected based on the window you set. 5) Log-level and indicators - Temporarily increase logs to debug in the services container: -

PREFECT_LOGGING_LEVEL=DEBUG

- Look for entries showing “reactive evaluation”/“proactive evaluation,” “trigger fired,” and “action executed” — if triggers fire immediately but actions execute later, the delay is in the actions path. 6) Database health - If using SQLite, this is very likely the cause — please switch to PostgreSQL. - If using PostgreSQL, ensure it has enough resources and connections, and that you’re not seeing lock waits/timeouts during peak times. Example Docker Compose tuning (services container)

services:
  prefect-services:
    environment:
      - PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
      - PREFECT_SERVER_SERVICES_FOREMAN_LOOP_SECONDS=5
      - PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_HEARTBEAT_INTERVAL_SECONDS=5
      - PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_RECONNECT_INTERVAL_SECONDS=10
      - PREFECT_SERVER_DATABASE_TIMEOUT=30
      - TZ=UTC

A few clarifying questions to help me pinpoint this for you: - What Prefect version are you on?

- Docker Compose or Kubernetes? If K8s, how many replicas of your “services” component are running? - Which database are you using (PostgreSQL vs SQLite)? - Are your slow automations reactive (event-driven like flow-run state changes) or proactive (absence-of-event for N minutes)? If possible, share one example of the automation’s condition/action. - Do you see trigger “fired” quickly in logs, with the action running much later, or is the trigger itself delayed? If you can share those details, I can suggest precise settings to tweak and where to inspect next.

Thanks — that detail helps a lot. Given: - Kubernetes - Two Prefect “server” instances - PostgreSQL - Automations are reactive - Event appears immediately in the feed, but the automation “fires” ~15 minutes later and the action executes immediately This pattern almost always means the reactive trigger evaluation is not receiving events in real time. Instead, a periodic/fallback sweep is picking them up later — and 10–15 minutes is a typical delay you’d see from a fallback path. Most common causes and fixes 1) Your “services” pod is not running or not healthy - ReactiveTriggers runs in the “services” component, not the API. On K8s, you should have two deployments: - API (can be scaled >1) - Services (should usually be 1 replica) - If the services pod is missing, CrashLooping, or cannot connect to Postgres, reactive automations won’t process events immediately — they’ll only be caught by a periodic sweep later. - Action: Ensure the services deployment is present and running exactly 1 replica and that

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

. Docs: - Prefect Helm (Kubernetes) - Services overview via Docker Compose (concepts carry over) 2) PgBouncer (or any DB proxy) in transaction pooling mode - Reactive triggers rely on PostgreSQL LISTEN/NOTIFY, which requires a persistent session connection. - If you connect to Postgres through PgBouncer in transaction pooling mode, LISTEN/NOTIFY will not work reliably — the listener won’t receive notifications, and your system will fall back to the slow, periodic sweep. - Action: If you use PgBouncer, switch the connection for the services pod to session pooling or bypass PgBouncer and connect directly to Postgres for the services pod. 3) Multiple services replicas - Running more than 1 replica of the services (triggers) can cause contention or odd behavior; it won’t usually cause a 15-minute lag, but it’s not recommended. - Action: Scale services to 1 replica. 4) Postgres connectivity and settings on the services pod - Ensure the services pod has the same DB URL as the API and can reach it. - Useful environment variables: -

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

-

PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_HEARTBEAT_INTERVAL_SECONDS=5

-

PREFECT_SERVER_SERVICES_TRIGGERS_PG_NOTIFY_RECONNECT_INTERVAL_SECONDS=10

-

PREFECT_SERVER_DATABASE_TIMEOUT=30

(or 60 if you see timeouts) - If the listener repeatedly disconnects/reconnects, you’ll see delays. Settings reference: - Settings reference - Automations and event triggers 5) Clock skew - If system clocks differ across pods/DB by minutes, reactive evaluation can appear late. - Action: Ensure NTP is enabled and all pods/nodes/DB are synced; set

TZ=UTC

on all containers. What to check next (quick checklist) - Do you have a “services” Deployment/Pod running in your cluster? Is it healthy? How many replicas? - Are you routing Postgres through PgBouncer or a proxy? If yes, is it using transaction pooling? If so, connect the services pod with session pooling or directly to Postgres. - Check services pod logs: - Look for lines indicating “ReactiveTriggers” started and “listening for pg_notify.” - Look for repeated reconnects or errors related to LISTEN/NOTIFY or DB locks/timeouts. - Verify effective config: - In any Prefect container/pod:

prefect config view --show-defaults

Confirm the triggers service is enabled and the DB timeout is reasonable. If you can share: - Whether you have a separate “services” deployment and how many replicas it has - Whether PgBouncer (or another proxy) sits between Prefect and Postgres, and its pooling mode - Any relevant errors/warnings from the services pod logs around triggers/pg_notify

…I can give you precise steps to fix. My bet is either (1) the services pod isn’t running/healthy or (2) PgBouncer is in transaction pooling mode, breaking LISTEN/NOTIFY.

if so, are you using redis as a messaging broker?

@Marvin - Does it only make sense to have seperate server and and background services if I have a Redis setup? Or can i use it with just one postgresql instance?

Short answer: it makes sense to run separate “API” and “services” pods even if you only have PostgreSQL. You do not need Redis for Prefect’s automations. Prefect’s reactive automations use PostgreSQL LISTEN/NOTIFY under the hood; the background “services” process consumes those notifications. Why separate API and services even without Redis - The API is stateless and can be scaled horizontally. - The background services (ReactiveTriggers, ProactiveTriggers, Scheduler, Foreman, etc.) should generally run as a single replica to avoid duplicate work and to ensure orderly evaluation. - Both pods just point at the same PostgreSQL instance. Recommended K8s pattern - prefect-api Deployment: scale replicas as needed. - prefect-services Deployment: 1 replica only. - Both use the same database connection URL and the same settings profile. - Ensure triggers are enabled in the services pod:

PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true

- If you are using PgBouncer, the services pod must use session pooling (or connect directly to Postgres) so LISTEN/NOTIFY works; transaction pooling will break real-time delivery and cause the kind of 10–15 minute lag you’re seeing. Good refs - Kubernetes/Helm deployment - Automations and event triggers - Settings reference If you’d like, I can share a minimal K8s layout checklist for the two Deployments and what to set on each.