https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am trying to improve performance/sc...
# ask-marvin
r
@Marvin I am trying to improve performance/scaleability on my self-hosted prefect server - Reading your guide on how to do this. Is it required to use the redis cache to make that work? I am considering just having two 2 API instances and connection pooling and then leaving out the redis cache (potentially implementing it later on). Is that a viable solution, and if so - what would that solution look like?
m
✅ thought for 352.3 seconds
Short answer: yes—you can scale a self‑hosted Prefect 3 Server by running multiple stateless API instances behind a load balancer without Redis. Start with Postgres + 2 API replicas + connection pooling and a single “services” process. Add Redis later if/when you want cross‑instance real‑time streaming and event fan‑out. Key points and tradeoffs - API is stateless and horizontally scalable. Multiple replicas can share the same Postgres DB behind a load balancer. - You should run exactly one set of background services (scheduler, late-runs, repossessor, etc.) in the cluster to avoid double work. - Without Redis: - Orchestration, scheduling, workers polling for work, concurrency limits, logs ingestion, artifacts, etc. all work. - Real-time streaming across API replicas (UI “live” updates, server event streams, some automations/even ts fan‑out patterns) will be limited because the default in-process messaging broker is per-process. You can still query the API for data; it’s the cross-replica “push/stream” that’s affected. - With Redis (later): - Set the server’s events messaging broker/cache to Redis so event/log stream subscriptions and producers on different replicas coordinate cleanly. - No DB migration is required; you just add Redis and flip the settings, then roll the pods/containers. What a viable starting architecture looks like - Postgres (managed preferred). Optionally front with PgBouncer for DB connection pooling. - 2x API replicas behind a load balancer pointing to the same Postgres. - 1x Services process (singleton) for scheduler/foreman/etc. - Workers anywhere that point to the LB URL via PREFECT_API_URL. Example with Docker Compose (no Redis yet)
Copy code
version: "3.9"

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: prefect
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: prefect
    ports:
      - "5432:5432"

  # API replica 1
  prefect-api-1:
    image: prefecthq/prefect:3-latest
    command: >
      prefect server start
      --no-services
      --host 0.0.0.0
      --port 4200
    environment:
      PREFECT_SERVER_DATABASE__CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      # Tune SQLAlchemy pool to your load profile
      PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_SIZE: "20"
      PREFECT_SERVER_DATABASE__SQLALCHEMY__MAX_OVERFLOW: "20"
      PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_RECYCLE: "300"
    depends_on:
      - postgres

  # API replica 2
  prefect-api-2:
    image: prefecthq/prefect:3-latest
    command: >
      prefect server start
      --no-services
      --host 0.0.0.0
      --port 4200
    environment:
      PREFECT_SERVER_DATABASE__CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
      PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_SIZE: "20"
      PREFECT_SERVER_DATABASE__SQLALCHEMY__MAX_OVERFLOW: "20"
      PREFECT_SERVER_DATABASE__SQLALCHEMY__POOL_RECYCLE: "300"
    depends_on:
      - postgres

  # Singleton background services (scheduler, etc.)
  prefect-services:
    image: prefecthq/prefect:3-latest
    command: prefect server services start
    environment:
      PREFECT_SERVER_DATABASE__CONNECTION_URL: <postgresql+asyncpg://prefect:prefect@postgres:5432/prefect>
    depends_on:
      - postgres
- Put a reverse proxy / load balancer in front of 
prefect-api-1
and 
prefect-api-2
(e.g., NGINX, ALB, etc). - Point all clients/workers at the LB: - 
PREFECT_API_URL=<https://your-lb.example.com/api>
Kubernetes outline - Deployment: 
prefect-api
with replicas: 2 - Container command: 
prefect server start --no-services --host 0.0.0.0 --port 4200
- Env: DB connection URL and SQLAlchemy pool tuning - Service + Ingress/ALB in front - Deployment: 
prefect-services
replicas: 1 - Command: 
prefect server services start
- Env: DB connection URL - All workers and CLI clients use 
PREFECT_API_URL
pointing at the 
prefect-api
Service/Ingress. When to add Redis and how
- Symptoms that suggest enabling Redis: - Live UI updates intermittently miss messages or only reflect changes from the API instance the browser hit. - You split services across more than one process and want robust event/log fan‑out between them and the API replicas. - Add a Redis instance and set the server events broker/cache settings to Redis (roll your API/Services deployments afterward). See the Redis integration for the exact configuration knobs: - Prefect Redis integration - Settings reference (search for ServerEvents/messaging broker/cache) - No database changes required; this is a runtime configuration switch. Tuning notes - DB connections: Each API replica maintains its own SQLAlchemy pool. Tune pool_size/max_overflow/recycle based on expected parallel API traffic. With many replicas, a PgBouncer in transaction pooling mode between API and Postgres can help. - API process count: Prefer scaling via more pods/containers rather than turning up multiple uvicorn workers inside one pod. - Workers: Poll the API via HTTP; they do not require Redis to fetch work. - Optional: If you notice websocket streaming quirks in your LB, enable sticky sessions only for websocket paths to keep a browser UI session on one API instance. Docs you can reference - Server concepts: Prefect Server concepts - Self-hosted reference (Compose): Self‑hosted with Docker Compose - Settings reference: Prefect settings reference - Redis integration: Prefect Redis integration If you share your target scale (concurrent flow runs/requests) and where you’re deploying (Compose vs. Kubernetes), I can give more concrete pool sizes and resource suggestions.
r
@Marvin - I have implemented the suggested solution for scaling a self hosted prefect server (seperate api and background scheduler and Redis cache). Most of it seems to be working fine, however my automations are no longer executing. I can see in the event feed that automations are triggered, however they are never executed. What might the reason for this?
m
✅ thought for 293.3 seconds
Got it — when automations “trigger” in the event feed but no actions happen, it’s almost always one of these: Most common causes - Triggers service not running: The background “triggers” services (reactive/proactive) are what actually evaluate automations and execute actions. If they’re not running (or are disabled), you’ll see events but no actions. - Messaging backend mismatch between API and services: If your API instances publish events to Redis but your services process is still using the default in‑memory broker, the triggers service won’t see the events and no actions will fire. - prefect-redis not installed in all components: If the services container doesn’t have the prefect-redis integration installed, it cannot load the Redis messaging broker. - Actions can’t be executed (e.g., RunDeployment without a live worker): The automation may be evaluated but the action cannot create a run (wrong deployment name/work pool or no worker online). You won’t see new runs in this case. Quick diagnostics checklist 1) Confirm services are running (especially “triggers”) - On the box/container where you run background services:
Copy code
prefect server services ls
You should see entries for triggers (reactive/proactive), event_persister, event_logger, scheduler, etc. If they aren’t running:
Copy code
prefect server services start
Tail logs there to catch errors (e.g., broker import errors). 2) Verify both API and services use the same messaging backend (Redis) - In Prefect 3.x the defaults are in‑memory: - 
server.events.messaging_broker = prefect.server.utilities.messaging.memory
- 
server.events.messaging_cache   = prefect.server.utilities.messaging.memory
- In your deployment, set BOTH to the Redis implementation from the prefect-redis integration and ensure the same settings are present in: - all API replicas - the background services process - You can confirm the server’s effective settings by hitting the admin settings endpoint on any API instance:
Copy code
GET /api/admin/settings
Look for 
server.events.messaging_broker
and 
server.events.messaging_cache
. They must point to the Redis messaging modules (not memory) and match between API and services. - Also ensure the Redis connection URL you’re using is reachable from both API and services. 3) Ensure prefect-redis is installed everywhere - The API image(s) and the services image must both have the integration:
Copy code
pip install prefect-redis
If it’s missing in the services container, you’ll see import errors or it will silently fall back to memory. 4) Check automation action feasibility - If your automation uses RunDeployment: - Confirm the deployment slug exists and matches exactly. - Confirm the target work pool exists and at least one worker is online; otherwise a run will not be picked up. - If the action is something else (e.g., SendNotification), check for errors in the services logs. 5) Look for errors in services logs - The triggers/event_persister services will log if they cannot connect to Redis, can’t import the broker, or fail an action. If you can share: - The output of 
prefect server services ls
- The relevant pieces of your env/config showing the values you set for: - 
PREFECT_SERVER_EVENTS__MESSAGING_BROKER
- 
PREFECT_SERVER_EVENTS__MESSAGING_CACHE
- your Redis URL/config - Whether the automation action is RunDeployment (and the target work pool/worker status) …I can pinpoint the misconfig quickly. References - Server concepts: Prefect Server concepts - Settings reference (events messaging): Prefect settings reference - Redis integration: prefect-redis integration
r
@Marvin - Below is prefect server services: ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Name ┃ Enabled? ┃ Description ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ CancellationCleanup │ ✓ PREFECT_SERVER_SERVICES_CANCELLATION_CLEANUP_ENABLED │ Cancels tasks and subflows of │ │ │ │ flow runs that have been │ │ │ │ cancelled │ │ Foreman │ ✓ PREFECT_SERVER_SERVICES_FOREMAN_ENABLED │ Monitors the status of workers │ │ │ │ and their associated work │ │ │ │ pools │ │ MarkLateRuns │ ✓ PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED │ Finds flow runs that are later │ │ │ │ than their scheduled start │ │ │ │ time │ │ FailExpiredPauses │ ✓ PREFECT_SERVER_SERVICES_PAUSE_EXPIRATIONS_ENABLED │ Fails flow runs that have been │ │ │ │ paused and never resumed │ │ Repossessor │ ✓ PREFECT_SERVER_SERVICES_REPOSSESSOR_ENABLED │ Handles the reconciliation of │ │ │ │ expired leases; no tow truck │ │ │ │ dependency. │ │ RecentDeploymentsScheduler │ ✓ PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED │ Schedules deployments that │ │ │ │ were updated very recently │ │ Scheduler │ ✓ PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED │ Schedules flow runs from │ │ │ │ deployments. │ │ TaskRunRecorder │ ✓ PREFECT_SERVER_SERVICES_TASK_RUN_RECORDER_ENABLED │ Constructs task runs and │ │ │ │ states from client-emitted │ │ │ │ events │ │ Telemetry │ ✓ PREFECT_SERVER_ANALYTICS_ENABLED │ Sends anonymous data to │ │ │ │ Prefect to help us improve │ │ EventLogger │ x PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED │ A debugging service that logs │ │ │ │ events to the console as they │ │ │ │ arrive. │ │ EventPersister │ ✓ PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED │ A service that persists events │ │ │ │ to the database as they │ │ │ │ arrive. │ │ ProactiveTriggers │ ✓ PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED │ Evaluates proactive automation │ │ │ │ triggers │ │ ReactiveTriggers │ ✓ PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED │ Evaluates reactive automation │ │ │ │ triggers │ │ Actions │ ✓ PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED │ Runs the actions triggered by │ │ │ │ automations │ │ Distributor │ ✓ PREFECT_API_EVENTS_STREAM_OUT_ENABLED │ A marker class for services │ │ │ │ that should run even when │ │ │ │ running an ephemeral server │ │ LogDistributor │ x PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED │ Service for distributing logs │ │ │ │ to websocket subscribers │ └────────────────────────────┴────────────────────────────────────────────────────────┴────────────────────────────────┘
m
🔄 Researching your answer... this may take a while 🔧 Using tool: 
check_cli_command
📊 Tools used: • 
check_cli_command
(1x)
Thanks for sharing the services table — that’s the output of 
prefect server services ls
in Prefect 3.x. Here’s how to interpret and control these services, plus links to docs and CLI help. What each service does (quick guide) - Scheduler: Schedules flow runs from deployments. - RecentDeploymentsScheduler: Accelerates scheduling for deployments updated very recently. - Foreman: Monitors workers and their work pools. - CancellationCleanup: Cancels tasks/subflows of cancelled flow runs. - MarkLateRuns: Marks runs late if they’re past their scheduled start time. - FailExpiredPauses: Fails runs that were paused but never resumed before expiration. - Repossessor: Reconciles expired leases. - TaskRunRecorder: Builds task runs and states from client-emitted events. - Triggers (Proactive/Reactive/Actions): Evaluates automations and runs their actions. - EventLogger: Debug service that logs incoming events to the server console (disabled by default to avoid noise). - EventPersister: Persists incoming events to the DB. - LogDistributor: Streams logs to websocket subscribers (disabled by default). - Distributor: Marker service for things that should run in ephemeral mode. How to enable/disable - Toggle each with an environment variable using the pattern shown in your table: 
PREFECT_SERVER_SERVICES_<SERVICE>_ENABLED=true|false
- For log streaming specifically, use 
PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED=true
(controls the LogDistributor). - Changes take effect when the services process (or container) restarts. Examples
Copy code
# Enable EventLogger (prints events to server logs; noisy but useful for debugging)
PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true

# Enable log streaming over websockets (for live logs)
PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED=true

# Disable LateRuns service
PREFECT_SERVER_SERVICES_LATE_RUNS_ENABLED=false
Docker Compose snippet (if you’re running the “prefect-services” container)
Copy code
services:
  prefect-services:
    environment:
      - PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true
      - PREFECT_SERVER_LOGS_STREAM_OUT_ENABLED=true
CLI you can use - List services/status:
Copy code
prefect server services ls
- Start all enabled services in one process:
Copy code
prefect server services start
- Stop background services started by the command above:
Copy code
prefect server services stop
Docs and references - Server services CLI: CLI reference: prefect server services - Settings reference (all env vars + defaults): Settings reference - Scheduler: Scheduler service - Foreman: Foreman service - Late runs: LateRuns service - Cancellation cleanup: CancellationCleanup service - Pause expirations: FailExpiredPauses service - Repossessor: Repossessor service - Event logger: EventLogger service - Event persister: EventPersister service - Triggers: Triggers services - Log distributor: LogDistributor - Self-hosted guide (Compose): Self-hosted with Docker Compose
If you tell me which behavior you want to change (e.g., enable live log streaming to the UI, disable a service, debug events), I can give exact env vars and where to set them for your setup.
r
@Marvin - I have a problem with my prefect setup (2 apis, 1 background scheduler and a redis cache) running in k8s. My automations are being triggered, but they are never executed
m
✅ thought for 49.7 seconds
Got it — when automations are “triggered” but nothing actually happens, it’s usually because the server “Actions” runner isn’t running anywhere, or the action created a flow run that no worker picked up. Let’s narrow it down. Key context - In Prefect 3.x, automations (triggers + actions) are executed by the Triggers/Actions server service — not by the API or the Scheduler. - Your setup (“2 APIs, 1 background scheduler, redis”) sounds like you are not running the Triggers/Actions service. In that case, triggers can be evaluated (and appear as “Triggered”), but actions never execute. Quick checks 1) Confirm the services actually running - From inside your “services”/background pod (if you have one), run:
Copy code
prefect server services ls
- You should see: - ProactiveTriggers / ReactiveTriggers / Actions: Enabled ✓ - EventPersister: Enabled ✓ - Scheduler: Enabled ✓ - Foreman: Enabled ✓ (helps monitor workers/pools) 2) If you don’t have a “services” process - Start one as a dedicated K8s Deployment (example below) that runs all enabled services:
Copy code
prefect server services start
- Ensure env vars include:
Copy code
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
3) If your automation action is “Run deployment” - Check the UI to see if a flow run is being created when the automation fires. - If a flow run exists but stays Scheduled, that means no worker picked it up: - Verify the target work pool/queue is correct and not paused. - Ensure a worker is connected to the right API URL and polling that pool. - Check worker logs or 
prefect worker inspect
for Last polled times. - Foreman being healthy helps detect “no worker polling” cases. 4) Look for errors in logs - Services pod logs should show the Triggers/Actions services starting and any action execution messages. - API pod logs may show action creation attempts or errors. - If you rely on events for trigger conditions, EventPersister must be running. Minimal K8s Deployment for services (add alongside your API)
Copy code
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-services
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-services
  template:
    metadata:
      labels:
        app: prefect-services
    spec:
      containers:
        - name: services
          image: prefecthq/prefect:3-latest
          command: ["sh","-lc","prefect server services start"]
          env:
            - name: PREFECT_API_URL
              value: "http://<your-api-service>:4200/api"
            # Ensure triggers and event persister are on
            - name: PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED
              value: "true"
            - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED
              value: "true"
            # You can also enable Scheduler here (or keep it in your existing scheduler pod)
            - name: PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED
              value: "true"
If you already have a services pod, please share: - Prefect version - The automation’s action type(s) - Output of 
prefect server services ls
from that pod - Whether a flow run is created when the automation fires (and its state) - Any relevant logs mentioning “triggers” or “actions” Helpful references - CLI for services: prefect server services CLI - Settings reference (env vars): Settings reference - Triggers/Actions service: Triggers services - Event persister: EventPersister - Scheduler: Scheduler - Foreman: Foreman
My hunch: you’ve got the Scheduler running but not the Triggers/Actions service. Add the “services” process (or enable triggers in your existing services pod) and your automations should start executing. If it’s a worker pickup issue, we’ll pin that down with the flow run/worker checks above.
r
@Marvin I get these logs from my background: 063144.142 | INFO | prefect.server.events.triggers - Automation f3260518-7304-4960-95ec-d79af071e003, trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 covers event 'prefect.flow-run.Completed' (019a95a9-7c4c-7966-bb66-6a22653dc133) for 'prefect.flow-run.019a9563-8a65-736f-ac0b-28c4a62cf941' at '2025-11-18T063143.948301+00:00' 063144.470 | DEBUG | prefect.server.events.triggers - Automation f3260518-7304-4960-95ec-d79af071e003 ('teams-notification-flow-run-failed-severity-critical') trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 triggered for keys (['prefect.flow-run.019a9563-8a65-736f-ac0b-28c4a62cf941']) reactively, having occurred 1 times between 2025-11-18 061643.948301+00:00 and 2025-11-18 063143.948301+00:00 063144.473 | DEBUG | prefect.server.events.messaging - Publishing event: prefect.automation.triggered with id: 019a95a9-7e57-7cfa-855b-78af099bd634 for resource: prefect.automation.f3260518-7304-4960-95ec-d79af071e003 063144.502 | DEBUG | prefect.server.events.services.event_persister - Received event: prefect.automation.triggered with id: 019a95a9-7e57-7cfa-855b-78af099bd634 for resource: prefect.automation.f3260518-7304-4960-95ec-d79af071e003 063144.954 | INFO | prefect.server.events.triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14, trigger 622a9503-85aa-49b7-b313-d36d9586cf89 covers event 'prefect.flow-run.Completed' (019a95a9-7c4c-7966-bb66-6a22653dc133) for 'prefect.flow-run.019a9563-8a65-736f-ac0b-28c4a62cf941' at '2025-11-18T063143.948301+00:00' 063145.205 | DEBUG | prefect.server.events.triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14 ('test') trigger 622a9503-85aa-49b7-b313-d36d9586cf89 triggered for keys (['prefect.flow-run.019a9563-8a65-736f-ac0b-28c4a62cf941']) reactively, having occurred 1 times between 2025-11-18 061643.948301+00:00 and 2025-11-18 063143.948301+00:00 063145.208 | DEBUG | prefect.server.events.messaging - Publishing event: prefect.automation.triggered with id: 019a95a9-8136-7865-92c9-1a2ebcdc262f for resource: prefect.automation.a8aebf97-183f-4600-a46e-1f7b57880f14 063145.223 | DEBUG | prefect.server.events.services.event_persister - Received event: prefect.automation.triggered with id: 019a95a9-8136-7865-92c9-1a2ebcdc262f for resource: prefect.automation.a8aebf97-183f-4600-a46e-1f7b57880f14 063153.534 | INFO | prefect.server.events.triggers - Automation f3260518-7304-4960-95ec-d79af071e003, trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 covers event 'prefect.flow-run.Completed' (019a95a9-a0a1-75f4-b555-cb7f75ed8748) for 'prefect.flow-run.019a9563-83ea-7270-86a9-b7bb593f2ab0' at '2025-11-18T063153.249920+00:00' 063153.805 | DEBUG | prefect.server.events.triggers - Automation f3260518-7304-4960-95ec-d79af071e003 ('teams-notification-flow-run-failed-severity-critical') trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 triggered for keys (['prefect.flow-run.019a9563-83ea-7270-86a9-b7bb593f2ab0']) reactively, having occurred 1 times between 2025-11-18 061653.249920+00:00 and 2025-11-18 063153.249920+00:00 063153.808 | DEBUG | prefect.server.events.messaging - Publishing event: prefect.automation.triggered with id: 019a95a9-a2ce-7355-92a9-416f73724833 for resource: prefect.automation.f3260518-7304-4960-95ec-d79af071e003 063153.814 | DEBUG | prefect.server.events.services.event_persister - Received event: prefect.automation.triggered with id: 019a95a9-a2ce-7355-92a9-416f73724833 for resource: prefect.automation.f3260518-7304-4960-95ec-d79af071e003 063154.279 | INFO | prefect.server.events.triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14, trigger 622a9503-85aa-49b7-b313-d36d9586cf89 covers event 'prefect.flow-run.Completed' (019a95a9-a0a1-75f4-b555-cb7f75ed8748) for 'prefect.flow-run.019a9563-83ea-7270-86a9-b7bb593f2ab0' at '2025-11-18T063153.249920+00:00' 063154.767 | DEBUG | prefect.server.events.triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14 ('test') trigger 622a9503-85aa-49b7-b313-d36d9586cf89 triggered for keys (['prefect.flow-run.019a9563-83ea-7270-86a9-b7bb593f2ab0']) reactively, having occurred 1 times between 2025-11-18 061653.249920+00:00 and 2025-11-18 063153.249920+00:00 063154.771 | DEBUG | prefect.server.events.messaging - Publishing event: prefect.automation.triggered with id: 019a95a9-a690-708e-a3a6-aa8a8781e396 for resource: prefect.automation.a8aebf97-183f-4600-a46e-1f7b57880f14 063154.821 | DEBUG | prefect.server.events.services.event_persister - Received event: prefect.automation.triggered with id: 019a95a9-a690-708e-a3a6-aa8a8781e396 for resource: prefect.automation.a8aebf97-183f-4600-a46e-1f7b57880f14
m
Your message was too long, here's your message at the allowed limit: I get these logs from my background 06 31 44 142 | INFO | prefect server events triggers - Automation f3260518-7304-4960-95ec-d79af071e003 trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 covers event 'prefect flow-run Completed' (019a95a9-7c4c-7966-bb66-6a22653dc133) for 'prefect flow-run 019a9563-8a65-736f-ac0b-28c4a62cf941' at '2025-11-18T06 31 43 948301+00 00' 06 31 44 470 | DEBUG | prefect server events triggers - Automation f3260518-7304-4960-95ec-d79af071e003 ('teams-notification-flow-run-failed-severity-critical') trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 triggered for keys (['prefect flow-run 019a9563-8a65-736f-ac0b-28c4a62cf941']) reactively having occurred 1 times between 2025-11-18 06 16 43 948301+00 00 and 2025-11-18 06 31 43 948301+00 00 06 31 44 473 | DEBUG | prefect server events messaging - Publishing event prefect automation triggered with id 019a95a9-7e57-7cfa-855b-78af099bd634 for resource prefect automation f3260518-7304-4960-95ec-d79af071e003 06 31 44 502 | DEBUG | prefect server events services event_persister - Received event prefect automation triggered with id 019a95a9-7e57-7cfa-855b-78af099bd634 for resource prefect automation f3260518-7304-4960-95ec-d79af071e003 06 31 44 954 | INFO | prefect server events triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14 trigger 622a9503-85aa-49b7-b313-d36d9586cf89 covers event 'prefect flow-run Completed' (019a95a9-7c4c-7966-bb66-6a22653dc133) for 'prefect flow-run 019a9563-8a65-736f-ac0b-28c4a62cf941' at '2025-11-18T06 31 43 948301+00 00' 06 31 45 205 | DEBUG | prefect server events triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14 ('test') trigger 622a9503-85aa-49b7-b313-d36d9586cf89 triggered for keys (['prefect flow-run 019a9563-8a65-736f-ac0b-28c4a62cf941']) reactively having occurred 1 times between 2025-11-18 06 16 43 948301+00 00 and 2025-11-18 06 31 43 948301+00 00 06 31 45 208 | DEBUG | prefect server events messaging - Publishing event prefect automation triggered with id 019a95a9-8136-7865-92c9-1a2ebcdc262f for resource prefect automation a8aebf97-183f-4600-a46e-1f7b57880f14 06 31 45 223 | DEBUG | prefect server events services event_persister - Received event prefect automation triggered with id 019a95a9-8136-7865-92c9-1a2ebcdc262f for resource prefect automation a8aebf97-183f-4600-a46e-1f7b57880f14 06 31 53 534 | INFO | prefect server events triggers - Automation f3260518-7304-4960-95ec-d79af071e003 trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 covers event 'prefect flow-run Completed' (019a95a9-a0a1-75f4-b555-cb7f75ed8748) for 'prefect flow-run 019a9563-83ea-7270-86a9-b7bb593f2ab0' at '2025-11-18T06 31 53 249920+00 00' 06 31 53 805 | DEBUG | prefect server events triggers - Automation f3260518-7304-4960-95ec-d79af071e003 ('teams-notification-flow-run-failed-severity-critical') trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 triggered for keys (['prefect flow-run 019a9563-83ea-7270-86a9-b7bb593f2ab0']) reactively having occurred 1 times between 2025-11-18 06 16 53 249920+00 00 and 2025-11-18 06 31 53 249920+00 00 06 31 53 808 | DEBUG | prefect server events messaging - Publishing event prefect automation triggered with id 019a95a9-a2ce-7355-92a9-416f73724833 for resource prefect automation f3260518-7304-4960-95ec-d79af071e003 06 31 53 814 | DEBUG | prefect server events services event_persister - Received event prefect automation triggered with id 019a95a9-a2ce-7355-92a9-416f73724833 for resource prefect automation f3260518-7304-4960-95ec-d79af071e003 06 31 54 279 | INFO | prefect server events triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14 trigger 622a9503-85aa-49b7-b313-d36d9586cf89 covers event 'prefect flow-run Completed' (019a95a9-a0a1-75f4-b555-cb7f75ed8748) for 'prefect flow-run 019a9563-83ea-7270-86a9-b7bb593f2ab0' at '2025-11-18T06 31 53 249920+00 00' 06 31 54 767 | DEBUG | prefect server events triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880
f14 ('test') trigger 622a9503-85aa-49b7-b313-d36d9586cf89 triggered for keys (['prefect flow-run 019a9563-83ea-7270-86a9-b7bb593f2ab0']) reactively having occurred 1 times between 2025-11-18 06 16 53 249920+00 00 and 2025-11-18 06 31 53 249920+00 00 06 31 54 771 | DEBUG | prefect server events messaging - Publishing event prefect automation triggered with id 019a95a9-a690-708e-a3a6-aa8a8781e396 for resource prefect automation a8aebf97-183f-4600-a46e-1f7b57880f14 06 31 54 821 | DEBUG | prefect server events services event_persister - Received event prefect automation triggered with
r
@Marvin I get these logs: kubectl logs -n prefect prefect-server-background-76fbf85f6c-p9hwz --tail=100 | grep -i "automation" 063406.200 | DEBUG | prefect.server.events.triggers - Automation f3260518-7304-4960-95ec-d79af071e003 ('teams-notification-flow-run-failed-severity-critical') trigger 7a18917b-0a7d-421f-ab99-2d047d5a76c9 triggered for keys (['prefect.flow-run.019a9563-89c7-70f8-be9a-efd490fef6e6']) reactively, having occurred 1 times between 2025-11-18 061905.437997+00:00 and 2025-11-18 063405.437997+00:00 063406.203 | DEBUG | prefect.server.events.messaging - Publishing event: prefect.automation.triggered with id: 019a95ab-a7f9-7263-871b-37d9dcc7a3df for resource: prefect.automation.f3260518-7304-4960-95ec-d79af071e003 063406.271 | DEBUG | prefect.server.events.services.event_persister - Received event: prefect.automation.triggered with id: 019a95ab-a7f9-7263-871b-37d9dcc7a3df for resource: prefect.automation.f3260518-7304-4960-95ec-d79af071e003 063406.779 | INFO | prefect.server.events.triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14, trigger 622a9503-85aa-49b7-b313-d36d9586cf89 covers event 'prefect.flow-run.Completed' (019a95ab-a4fd-763e-a6f0-0546a648e7a9) for 'prefect.flow-run.019a9563-89c7-70f8-be9a-efd490fef6e6' at '2025-11-18T063405.437997+00:00' 063407.453 | DEBUG | prefect.server.events.triggers - Automation a8aebf97-183f-4600-a46e-1f7b57880f14 ('test') trigger 622a9503-85aa-49b7-b313-d36d9586cf89 triggered for keys (['prefect.flow-run.019a9563-89c7-70f8-be9a-efd490fef6e6']) reactively, having occurred 1 times between 2025-11-18 061905.437997+00:00 and 2025-11-18 063405.437997+00:00 063407.458 | DEBUG | prefect.server.events.messaging - Publishing event: prefect.automation.triggered with id: 019a95ab-acdf-7586-a1b7-2e1440fd9c24 for resource: prefect.automation.a8aebf97-183f-4600-a46e-1f7b57880f14 063407.519 | DEBUG | prefect.server.events.services.event_persister - Received event: prefect.automation.triggered with id: 019a95ab-acdf-7586-a1b7-2e1440fd9c24 for resource: prefect.automation.a8aebf97-183f-4600-a46e-1f7b57880f14
m
🔄 Researching your answer... this may take a while 🔧 Using tool: 
check_cli_command
📊 Tools used: • 
check_cli_command
(4x) • 
explore_module_offerings
(1x) • 
read_github_issues
(1x)
Thanks for the logs — they show your automations are being detected and “triggered,” and the events are being published and persisted. What’s missing are any signs that the actions actually ran. Most common causes - The Actions runner isn’t running (it’s part of the Triggers service bundle). - The automation only has “actions on resolve” (none on trigger). - The action failed (e.g., missing block, network egress, auth) but you’re grepping out the failure logs. - For RunDeployment actions: a flow run was created but no worker picked it up. Targeted checks 1) Confirm the Triggers/Actions service is running in your background pod
Copy code
prefect server services ls
You should see ProactiveTriggers, ReactiveTriggers, and Actions as Enabled ✓. 2) Inspect the automation to confirm it has actions_on_trigger
Copy code
prefect automation ls
prefect automation inspect <automation-id>
Look for actions_on_trigger vs actions_on_resolve. 3) Look explicitly for action execution/failure logs and events Remove the grep or use a broader one:
Copy code
# In the background/services pod
kubectl logs -n prefect <your-services-pod> --since=30m | \
  grep -E "events\.actions|automation\.action|ActionFailed|ActionSucceeded|SendNotification|RunDeployment"
You should see messages from prefect.server.events.actions and events like: - prefect.automation.action.attempted - prefect.automation.action.succeeded - prefect.automation.action.failed 4) Validate config in the services pod
Copy code
prefect config view --show-defaults --show-sources | \
  grep -E "PREFECT_SERVER_SERVICES_TRIGGERS|PREFECT_API_URL|PREFECT_API_DATABASE"
- Ensure 
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
- Ensure 
PREFECT_API_URL
points to your API service - Ensure the DB connection matches your API pods’ 5) If the action is SendNotification (your name suggests a Teams notification) - Confirm the MicrosoftTeamsWebhook block exists and the automation references it correctly (slug/name). - From the services pod, test basic egress to your Teams webhook host (to rule out egress or proxy issues). - If the block can’t be loaded or the POST fails, you’ll see an action.failed log. 6) If the action is RunDeployment - Check whether a flow run was created when the automation fired (UI or CLI):
Copy code
prefect flow-run ls --limit 20
- If created but stuck Scheduled: - Verify the target work pool/queue and that it’s not paused. - Ensure a worker is polling that pool (right API URL, last polled time). - Check 
prefect work-pool inspect <name>
and worker logs. Optionally enable extra diagnostics - Temporarily enable the event logger to see events in the services pod:
Copy code
PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true
If you can share: - Prefect version (
prefect version
) - Output of 
prefect server services ls
from the background pod - 
prefect automation inspect <id>
for one failing automation - Any lines that match the “action” grep above …I can pinpoint whether it’s a missing Actions runner, a misconfigured notification block, network egress, or a worker pickup issue. Docs for reference - CLI: prefect server services - Automations/actions internals: Actions API - Triggers service: Triggers/Actions service - Settings reference: Settings
r
@Marvin When i filter out just automation logs I only get logs indicating that automations are triggered. No logs indicating any errors or anything
m
✅ thought for 56.3 seconds
Thanks — grepping for “automation” will miss the logs from the Actions runner. The automations service logs under “events.actions” (and may not include the word “automation”). Let’s verify whether actions are actually being executed and whether the Actions runner is active. Please run these from the same background/services pod: 1) Confirm Actions/Triggers are enabled and running
Copy code
prefect server services ls
You should see ProactiveTriggers, ReactiveTriggers, and Actions as Enabled ✓. If Actions is not enabled, set:
Copy code
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
and restart the pod. Also verify the pod is started with:
Copy code
prefect server services start
2) Check settings in that pod
Copy code
prefect config view --show-defaults --show-sources | grep -E "PREFECT_SERVER_SERVICES_TRIGGERS|PREFECT_API_URL|PREFECT_LOGGING_SERVER_LEVEL"
- Ensure 
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
- Ensure 
PREFECT_API_URL
points at your API service - Consider raising verbosity: - 
PREFECT_LOGGING_SERVER_LEVEL=DEBUG
3) Search for action execution logs (broader than “automation”)
Copy code
# Last 30 minutes of logs; broaden patterns
kubectl logs -n prefect <your-services-pod> --since=30m | \
  grep -E "prefect\.server\.events\.actions|automation\.action|ActionFailed|ActionSucceeded|SendNotification|RunDeployment"
If actions are running, you should see events like: - prefect.automation.action.attempted - prefect.automation.action.succeeded - prefect.automation.action.failed 4) Inspect the automation to confirm it actually has actions_on_trigger
Copy code
prefect automation ls
prefect automation inspect f3260518-7304-4960-95ec-d79af071e003
prefect automation inspect a8aebf97-183f-4600-a46e-1f7b57880f14
Look for an action in 
actions_on_trigger
(e.g., SendNotification with a MicrosoftTeamsWebhook block). If there are only 
actions_on_resolve
, you’ll see triggers but no immediate action. 5) If it’s a Teams notification - Confirm the block exists with the expected name/slug. - Check egress from the services pod to your Teams webhook host (network/policy/proxy). - If it fails to send, you’ll get an action.failed log (which your current grep would miss). 6) If it’s a RunDeployment action - Check for new flow runs when the automation fires:
Copy code
prefect flow-run ls --limit 20
If a flow run is created but stays Scheduled, it’s a worker pickup issue: - Verify the work pool/queue is correct and not paused. - Ensure a worker is polling that pool, connected to the same API URL. If you share: - Output of 
prefect server services ls
from the background pod - The 
prefect automation inspect
output for one of these automations - The broader grep results above (or last 200 lines without grep) …I can pinpoint if the Actions runner isn’t active, if the automation lacks trigger actions, or if there’s a notification/block/egress or worker pickup issue. Refs - CLI: prefect server services - Actions internals: Actions API - Triggers service: Triggers/Actions service
25 Views
Open in Slack
PreviousNext
Powered by