https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I'm deploying a self-hosted Prefect s...
# ask-marvin
u
@Marvin I'm deploying a self-hosted Prefect server to Kubernetes as a background service, but I've discovered that Events are not functioning. The Event feed screen shows absolutely no events recorded. Please advise on troubleshooting steps. I can provide necessary information like the Pod Spec if needed.
m
✅ thought for 219.0 seconds
Thanks for the detail — we can definitely get this working. In Prefect 3.x self-hosted, Events are produced by the API and processed by background services in the API pod(s). If the Event feed is empty, it’s usually one of: - Event services not started (EventPersister, Distributor, Triggers, Actions) - WebSocket stream blocked by ingress/proxy - DB tables/migrations not present or persister can’t write - Multi-replica API without a shared messaging backend Below is a focused troubleshooting checklist you can run now, plus what info would help me help you faster. Quick checks 1) Confirm your Prefect version - Please share the Prefect version running in your API image. - If it’s not the latest 3.x, consider upgrading after we capture current logs to rule out already-fixed issues. 2) Verify event services started - In the API pod logs, you should see EventPersister, Triggers, Actions, and the Distributor start without errors.
Copy code
kubectl logs deploy/<your-prefect-api-deployment> | egrep -i "EventPersister|Triggers|Actions|Distributor|event"
If you don’t see these or see errors, share that snippet. 3) Check the database has event tables and whether any records exist - From your DB, verify tables and row counts:
Copy code
-- PostgreSQL example
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'public' AND table_name IN ('events', 'event_resources');

SELECT COUNT(*) FROM events;
If the tables are missing or count is 0 even after a test event (next step), the persister isn’t writing. 4) Emit a test event directly to the API - Port-forward or hit your service internally, then POST a simple event:
Copy code
# adjust host/port to your API Service or port-forward target
curl -X POST http://<api-host>:<port>/api/events \
  -H "Content-Type: application/json" \
  -d '[{
    "occurred": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'",
    "event": "diagnostic.test",
    "resource": {"prefect.resource.id": "diagnostic/test"}
  }]'
Expected: HTTP 204. Then re-check:
Copy code
SELECT COUNT(*) FROM events;
5) Query events via the API - If the feed UI is empty, see if the API can return them:
Copy code
curl -X POST http://<api-host>:<port>/api/events/filter \
  -H "Content-Type: application/json" \
  -d '{"filter": {"events": ["*"]}, "limit": 5}'
If this returns your test event, the server is capturing events; the issue is likely WebSocket/ingress. 6) Check the UI WebSocket connection - In your browser dev tools Network tab, locate a WebSocket to 
/api/events/out
. - If it fails/doesn’t connect: - Ensure 
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true
on the API pod(s). - Check your ingress/proxy upgrades WebSockets and doesn’t buffer/close them. Example NGINX location block for WebSockets:
Copy code
location /api/events/out {
  proxy_pass http://<prefect-api-service>;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_read_timeout 3600s;
  proxy_send_timeout 3600s;
}
7) Confirm key environment variables in the API pod(s) - These should be set (defaults are generally okay, but verifying helps):
Copy code
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
PREFECT_SERVER_SERVICES_ACTIONS_ENABLED=true
# Optional tuning:
# PREFECT_SERVER_EVENTS_RETENTION_PERIOD=604800
- If you’re running multiple API replicas or want durability across restarts, use a shared broker (Redis) instead of the default in-memory queue:
Copy code
# requires the prefect-redis extra installed in your image
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
# (Configure your Redis connection per your environment)
Note: With multiple API replicas and in-memory messaging, events can be dropped or only visible on a single pod; a shared broker avoids that. 8) Turn on debug to see event activity - Temporarily enable: ```
PREFECT_LOGGING_LEVEL=DEBUG PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true
Copy code
Then check logs for lines like “Received event” and “Persisting X events”.

If you’re behind an ingress/proxy
- Ensure your `PREFECT_API_URL` in the UI resolves to your public/API ingress URL.
- Verify WebSocket upgrade headers/timeouts as shown above.
- If TLS-terminating, make sure the proxy passes the correct scheme/host headers.

If nothing shows after these steps
- Please share:
  - Prefect version
  - How you deployed (Helm chart or custom manifests)
  - Number of API replicas
  - The API pod env vars related to `PREFECT_*EVENT*` and service enables
  - Ingress controller and any special config (snippets for `/api/events` and `/api/events/out`)
  - A few lines of API pod logs around startup mentioning EventPersister/Distributor
  - DB type/version and results of the SQL checks above

Optional: minimal Deployment env example
env: - name: PREFECT_LOGGING_LEVEL value: INFO - name: PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED value: "true" - name: PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED value: "true" - name: PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED value: "true" - name: PREFECT_SERVER_SERVICES_ACTIONS_ENABLED value: "true" # For single replica, in-memory is fine; for >1 replica, switch to Redis: # - name: PREFECT_SERVER_EVENTS_MESSAGING_BROKER # value: prefect_redis.messaging # - name: PREFECT_SERVER_EVENTS_MESSAGING_CACHE # value: prefect_redis.messaging # - name: REDIS_URL # value: redis://redis:6379/0 # example; match your setup ``` Share your pod spec and logs when you can, and I’ll pinpoint the next steps quickly.
u
@Marvin Prefect version: 3.6.5 provide start-up logs of background-services as follows:
Copy code
03:18:11.265 | INFO    | prefect.server.events.services.triggers - ReactiveTriggers starting with unique consumer name: reactive-triggers-prefect-server-background-services-56c458c47d-h4xwf-8-553a9173
03:18:11.275 | INFO    | prefect.server.events.actions - Starting action message handler
03:18:11.276 | INFO    | prefect.server.services.task_run_recorder - Starting periodically process followers task every 5.0 seconds
03:18:11.277 | INFO    | prefect.server.events.triggers - Starting automation change listener
Event: 019cb152 2026-03-03T01:32:08.233005+00:00  (-6,363.09) 
[prefect.deployment.ready] 
prefect.deployment.02222e6e-01a4-4c2f-9b0f-0733d1a131ee
03:18:11.551 | INFO    | prefect.server.utilities.postgres_listener - Successfully established raw asyncpg connection for LISTEN/NOTIFY to <http://prefect-db.cqvqerbpboys.ap-northeast-1.rds.amazonaws.com:5432/prefectdb|prefect-db.cqvqerbpboys.ap-northeast-1.rds.amazonaws.com:5432/prefectdb>
03:18:11.551 | INFO    | prefect.server.events.triggers - Listening for automation changes on prefect_automation_changes
03:18:11.665 | INFO    | prefect.server.utilities.postgres_listener - Listening on Postgres channel: prefect_automation_changes
03:18:11.773 | INFO    | prefect.server.services.recentdeploymentsscheduler - Scheduled 0 runs.
03:18:11.819 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
03:18:11.820 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
03:18:11.822 | INFO    | prefect.server.services.scheduler - Scheduled 0 runs.
03:18:11.843 | INFO    | prefect.server.services.cancellationcleanup - Finished cleaning up cancelled flow runs.
03:18:16.266 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
03:18:16.270 | INFO    | prefect.server.services.recentdeploymentsscheduler - Scheduled 0 runs.
03:18:16.273 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
03:18:21.267 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
03:18:21.267 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
03:18:21.270 | INFO    | prefect.server.services.recentdeploymentsscheduler - Scheduled 0 runs.
03:18:26.264 | INFO    | prefect.server.services.marklateruns - Finished monitoring for late runs.
03:18:26.267 | INFO    | prefect.server.services.failexpiredpauses - Finished monitoring for late runs.
03:18:26.273 | INFO    | prefect.server.services.recentdeploymentsscheduler - Scheduled 0 runs.
m
✅ thought for 47.7 seconds
Thanks — this helps. From your logs I see: - ReactiveTriggers and Actions started - Other schedulers running - But I do not see EventPersister or the outbound WebSocket distributor starting If EventPersister isn’t running, nothing will be written to the events tables; if the distributor/stream is disabled or blocked, the UI feed will look empty even if events exist. Here’s a concise plan to pinpoint and fix it. What likely needs attention - EventPersister service is disabled or not starting in your API/background-services pod - Outbound event stream is disabled (or blocked by ingress/proxy) - If you have multiple API/background pods, using the default in-memory broker can result in missing events unless you switch to a shared broker (e.g., Redis) Immediate checks 1) Confirm EventPersister and the outbound stream are enabled in the API/background pod - Check the env on the pod(s) and add if missing:
Copy code
PREFECT_SERVER_SERVICES_EVENT_PERSISTER_ENABLED=true
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true
PREFECT_SERVER_SERVICES_TRIGGERS_ENABLED=true
PREFECT_SERVER_SERVICES_ACTIONS_ENABLED=true
# Optional for debugging:
# PREFECT_LOGGING_LEVEL=DEBUG
# PREFECT_SERVER_SERVICES_EVENT_LOGGER_ENABLED=true
- After restarting, you should see a line similar to:
Copy code
prefect.server.events.services.event_persister - Starting EventPersister
2) If you run multiple replicas of the API/background-services - Use a shared broker (Redis) instead of the default in-memory queue to avoid lost/partitioned events:
Copy code
# Ensure your image includes the prefect-redis extra and configure Redis URL
PREFECT_SERVER_EVENTS_MESSAGING_BROKER=prefect_redis.messaging
PREFECT_SERVER_EVENTS_MESSAGING_CACHE=prefect_redis.messaging
# plus your Redis connection settings
3) Verify the DB has event tables and whether events are being recorded
Copy code
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'public' AND table_name IN ('events', 'event_resources');

SELECT COUNT(*) FROM events;
- If tables exist but count stays 0 after the test below, EventPersister isn’t writing. 4) Emit a test event and confirm it lands
Copy code
curl -X POST http://<api-host>:<port>/api/events \
  -H "Content-Type: application/json" \
  -d '[{
    "occurred": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'",
    "event": "diagnostic.test",
    "resource": {"prefect.resource.id": "diagnostic/test"}
  }]'
- Expect 204 response; then:
Copy code
SELECT COUNT(*) FROM events;
- Also try fetching via API:
Copy code
curl -X POST http://<api-host>:<port>/api/events/filter \
  -H "Content-Type: application/json" \
  -d '{"filter": {"events": ["*"]}, "limit": 5}'
5) Check the UI WebSocket to the event feed - In browser dev tools, confirm a WebSocket to 
/api/events/out
connects. If it fails: - Ensure 
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED=true
- Verify your ingress/proxy upgrades WebSockets. Example NGINX location:
Copy code
location /api/events/out {
  proxy_pass http://<prefect-api-service>;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_read_timeout 3600s;
  proxy_send_timeout 3600s;
}
6) Get targeted logs
Copy code
kubectl logs deploy/<your-prefect-api-or-background-deploy> | egrep -i "event_persister|event persister|events.services|Distributor|websocket|event logger|prefect.server.events"
- Share any errors or absence of EventPersister startup. If you can, please share: - The Deployment(s) Pod spec env for your API/background-services (redact secrets) - Number of replicas for those pods - Ingress controller and any relevant location blocks for 
/api/events
and 
/api/events/out
- Result of the SQL checks and the test-event POST - A few lines of logs around startup that would include EventPersister/Distributor
With those, I can give you exact changes to make. Based on your current logs, my first bet is: enable EventPersister in the background-services/API pod and ensure the WebSocket stream is enabled and passed through your ingress.
u
@Marvin provide the current full server settings as follows:
Copy code
home: /home/prefect/.prefect
profiles_path: /home/prefect/.prefect/profiles.toml
debug_mode: false
api: {
  "url": null,
  "auth_string": null,
  "key": null,
  "tls_insecure_skip_verify": false,
  "ssl_cert_file": null,
  "enable_http2": false,
  "request_timeout": 60
}
cli: {
  "colors": true,
  "prompt": null,
  "wrap_lines": true
}
client: {
  "max_retries": 5,
  "retry_jitter_factor": 0.2,
  "retry_extra_codes": [],
  "csrf_support_enabled": true,
  "custom_headers": {},
  "metrics": {
    "enabled": true,
    "port": 4201
  }
}
cloud: {
  "api_url": "<https://api.prefect.cloud/api>",
  "enable_orchestration_telemetry": true,
  "max_log_size": 25000,
  "ui_url": "<https://app.prefect.cloud>"
}
deployments: {
  "default_work_pool_name": null,
  "default_docker_build_namespace": null
}
experiments: {
  "warn": true,
  "plugins": {
    "enabled": false,
    "allow": [],
    "deny": [],
    "setup_timeout_seconds": 20,
    "strict": false,
    "safe_mode": false
  }
}
flows: {
  "default_retries": 0,
  "default_retry_delay_seconds": 0
}
internal: {
  "logging_level": "WARNING"
}
logging: {
  "level": "INFO",
  "config_path": "/home/prefect/.prefect/logging.yml",
  "extra_loggers": [],
  "log_prints": false,
  "colors": true,
  "markup": false,
  "to_api": {
    "enabled": true,
    "batch_interval": 2,
    "batch_size": 4000000,
    "max_log_size": 1000000,
    "when_missing_flow": "warn"
  }
}
results: {
  "default_serializer": "pickle",
  "persist_by_default": false,
  "default_storage_block": null,
  "local_storage_path": "/home/prefect/.prefect/storage"
}
runner: {
  "process_limit": 5,
  "poll_frequency": 10,
  "heartbeat_frequency": null,
  "server": {
    "enable": false,
    "host": "localhost",
    "port": 8080,
    "log_level": "ERROR",
    "missed_polls_tolerance": 2
  }
}
server: {
  "logging_level": "INFO",
  "analytics_enabled": false,
  "metrics_enabled": true,
  "log_retryable_errors": true,
  "register_blocks_on_start": true,
  "memoize_block_auto_registration": true,
  "memo_store_path": "/home/prefect/.prefect/memo_store.toml",
  "deployment_schedule_max_scheduled_runs": 50,
  "api": {
    "auth_string": "**********",
    "host": "0.0.0.0",
    "port": 4200,
    "base_path": "/api",
    "default_limit": 200,
    "keepalive_timeout": 70,
    "csrf_protection_enabled": false,
    "csrf_token_expiration": "PT1H",
    "cors_allowed_origins": "*",
    "cors_allowed_methods": "*",
    "cors_allowed_headers": "*"
  },
  "concurrency": {
    "lease_storage": "prefect_redis.lease_storage",
    "initial_deployment_lease_duration": 300,
    "maximum_concurrency_slot_wait_seconds": 30
  },
  "database": {
    "sqlalchemy": {
      "connect_args": {
        "application_name": null,
        "statement_cache_size": null,
        "prepared_statement_cache_size": null,
        "tls": {
          "enabled": false,
          "ca_file": null,
          "cert_file": null,
          "key_file": null,
          "check_hostname": true
        }
      },
      "pool_size": 5,
      "pool_recycle": 3600,
      "pool_timeout": 30,
      "max_overflow": 10
    },
    "connection_url": "**********",
    "driver": null,
    "host": null,
    "port": null,
    "user": null,
    "name": null,
    "password": null,
    "echo": false,
    "migrate_on_start": false,
    "timeout": 30,
    "connection_timeout": 5
  },
  "deployments": {
    "concurrency_slot_wait_seconds": 30
  },
  "docket": {
    "name": "prefect-server",
    "url": "<redis://prefect-redis-replication-master.prefect.svc.cluster.local:6379/0>"
  },
  "ephemeral": {
    "enabled": true,
    "startup_timeout_seconds": 20
  },
  "events": {
    "stream_out_enabled": true,
    "related_resource_cache_ttl": "PT5M",
    "maximum_labels_per_resource": 500,
    "maximum_related_resources": 100,
    "maximum_size_bytes": 1500000,
    "expired_bucket_buffer": "PT1M",
    "proactive_granularity": "PT5S",
    "retention_period": "P7D",
    "maximum_websocket_backfill": "PT15M",
    "websocket_backfill_page_size": 250,
    "messaging_broker": "prefect_redis.messaging",
    "messaging_cache": "prefect_redis.messaging",
    "causal_ordering": "prefect_redis.ordering",
    "maximum_event_name_length": 1024
  },
  "flow_run_graph": {
    "max_nodes": 10000,
    "max_artifacts": 10000
  },
  "logs": {
    "stream_out_enabled": false,
    "stream_publishing_enabled": false
  },
  "services": {
    "cancellation_cleanup": {
      "enabled": true,
      "loop_seconds": 20
    },
    "event_persister": {
      "enabled": true,
      "batch_size": 20,
      "read_batch_size": 1,
      "flush_interval": 5,
      "batch_size_delete": 10000
    },
    "event_logger": {
      "enabled": true
    },
    "foreman": {
      "enabled": true,
      "loop_seconds": 15,
      "inactivity_heartbeat_multiple": 3,
      "fallback_heartbeat_interval_seconds": 30,
      "deployment_last_polled_timeout_seconds": 60,
      "work_queue_last_polled_timeout_seconds": 60
    },
    "late_runs": {
      "enabled": true,
      "loop_seconds": 5,
      "after_seconds": "PT15S"
    },
    "scheduler": {
      "enabled": true,
      "loop_seconds": 60,
      "deployment_batch_size": 100,
      "max_runs": 100,
      "min_runs": 3,
      "max_scheduled_time": "P100D",
      "min_scheduled_time": "PT1H",
      "insert_batch_size": 500,
      "recent_deployments_loop_seconds": 5
    },
    "pause_expirations": {
      "enabled": true,
      "loop_seconds": 5
    },
    "repossessor": {
      "enabled": true,
      "loop_seconds": 15
    },
    "task_run_recorder": {
      "enabled": true,
      "read_batch_size": 1
    },
    "triggers": {
      "enabled": true,
      "read_batch_size": 1,
      "pg_notify_reconnect_interval_seconds": 10,
      "pg_notify_heartbeat_interval_seconds": 5
    }
  },
  "tasks": {
    "tag_concurrency_slot_wait_seconds": 30,
    "max_cache_key_length": 2000,
    "scheduling": {
      "max_scheduled_queue_size": 1000,
      "max_retry_queue_size": 100,
      "pending_task_timeout": "PT0S"
    }
  },
  "ui": {
    "enabled": true,
    "api_url": "<dummy>",
    "serve_base": "/",
    "static_directory": "/ui_build",
    "show_promotional_content": true
  }
}
tasks: {
  "refresh_cache": false,
  "default_no_cache": false,
  "disable_caching": false,
  "default_retries": 0,
  "default_retry_delay_seconds": 0,
  "default_persist_result": null,
  "runner": {
    "thread_pool_max_workers": null,
    "process_pool_max_workers": null
  },
  "scheduling": {
    "default_storage_block": null,
    "delete_failed_submissions": true
  }
}
testing: {
  "test_mode": false,
  "unit_test_mode": false,
  "unit_test_loop_debug": true,
  "test_setting": "FOO"
}
worker: {
  "heartbeat_seconds": 30,
  "query_seconds": 10,
  "prefetch_seconds": 10,
  "webserver": {
    "host": "0.0.0.0",
    "port": 8080
  }
}
ui_url: 
silence_api_url_misconfiguration: false
m
Your message was too long, here's your message at the allowed limit: provide the current full server settings as follows ```home /home/prefect/ prefect profiles_path /home/prefect/ prefect/profiles toml debug_mode false api { url null auth_string null key null tls_insecure_skip_verify false ssl_cert_file null enable_http2 false request_timeout 60 } cli { colors true prompt null wrap_lines true } client { max_retries 5 retry_jitter_factor 0 2 retry_extra_codes [] csrf_support_enabled true custom_headers {} metrics { enabled true port 4201 } } cloud { api_url <https //api prefect cloud/api> enable_orchestration_telemetry true max_log_size 25000 ui_url <https //app prefect cloud> } deployments { default_work_pool_name null default_docker_build_namespace null } experiments { warn true plugins { enabled false allow [] deny [] setup_timeout_seconds 20 strict false safe_mode false } } flows { default_retries 0 default_retry_delay_seconds 0 } internal { logging_level WARNING } logging { level INFO config_path /home/prefect/ prefect/logging yml extra_loggers [] log_prints false colors true markup false to_api { enabled true batch_interval 2 batch_size 4000000 max_log_size 1000000 when_missing_flow warn } } results { default_serializer pickle persist_by_default false default_storage_block null local_storage_path /home/prefect/ prefect/storage } runner { process_limit 5 poll_frequency 10 heartbeat_frequency null server { enable false host localhost port 8080 log_level ERROR missed_polls_tolerance 2 } } server { logging_level INFO analytics_enabled false metrics_enabled true log_retryable_errors true register_blocks_on_start true memoize_block_auto_registration true memo_store_path /home/prefect/ prefect/memo_store toml deployment_schedule_max_scheduled_runs 50 api { auth_string ***** host 0 0 0 0 port 4200 base_path /api default_limit 200 keepalive_timeout 70 csrf_protection_enabled false csrf_token_expiration PT1H cors_allowed_origins * cors_allowed_methods * cors_allowed_headers * } concurrency { lease_storage prefect_redis lease_storage initial_deployment_lease_duration 300 maximum_concurrency_slot_wait_seconds 30 } database { sqlalchemy { connect_args { application_name null statement_cache_size null prepared_statement_cache_size null tls { enabled false ca_file null cert_file null key_file null check_hostname true } } pool_size 5 pool_recycle 3600 pool_timeout 30 max_overflow 10 } connection_url ***** driver null host null port null user null name null password null echo false migrate_on_start false timeout 30 connection_timeout 5 } deployments { concurrency_slot_wait_seconds 30 } docket { name prefect-server url <redis //prefect-redis-replication-master prefect svc cluster local 6379/0> } ephemeral { enabled true startup_timeout_seconds 20 } events { stream_out_enabled true related_resource_cache_ttl PT5M maximum_labels_per_resource 500 maximum_related_resources 100 maximum_size_bytes 1500000 expired_bucket_buffer PT1M proactive_granularity PT5S retention_period P7D maximum_websocket_backfill PT15M websocket_backfill_page_size 250 messaging_broker prefect_redis messaging messaging_cache prefect_redis messaging causal_ordering prefect_redis ordering maximum_event_name_length 1024 } flow_run_graph { max_nodes 10000 max_artifacts 10000 } logs { stream_out_enabled false stream_publishing_enabled false } services { cancellation_cleanup { enabled true loop_seconds 20 } event_persister { enabled true batch_size 20 read_batch_size 1 flush_interval 5 batch_size_delete 10000 } event_logger { enabled true } foreman { enabled true loop_seconds 15 inactivity_heartbeat_multiple 3 fallback_heartbeat_interval_seconds 30 deployment_last_polled_timeout_seconds 60 work_queue_last_polled_timeout_seconds 60 } late_runs { enabled true loop_seconds 5 after_seconds PT15S } scheduler { enabled true loop_seconds 60 deployment_batch_size 100 max_runs 100 min_runs 3 max_scheduled_time P100D min_scheduled_time PT1H insert_batch_size 500 recent_deployments_loop
_seconds 5 } pause_expirations { enabled true loop_seconds 5 } repossessor { enabled true loop_seconds 15 } task_run_recorder { enabled true read_batch_size 1 } triggers { enabled true read_batch_size 1 pg_notify_reconnect_interval_seconds 10 pg_notify_heartbeat_interval_seconds 5 } } tasks { tag_concurrency_slot_wait_seconds 30 max_cache_key_length 2000 scheduling { max_scheduled_queue_size 1000 max_retry_queue_size 100 pending_task_timeout PT0S } } ui { enabled true api_url <dummy>
u
@Marvin current server settings partially as follows:
Copy code
home: /home/prefect/.prefect
profiles_path: /home/prefect/.prefect/profiles.toml
debug_mode: false
api: {
  "url": null,
  "auth_string": null,
  "key": null,
  "tls_insecure_skip_verify": false,
  "ssl_cert_file": null,
  "enable_http2": false,
  "request_timeout": 60
}
cli: {
  "colors": true,
  "prompt": null,
  "wrap_lines": true
}
client: {
  "max_retries": 5,
  "retry_jitter_factor": 0.2,
  "retry_extra_codes": [],
  "csrf_support_enabled": true,
  "custom_headers": {},
  "metrics": {
    "enabled": true,
    "port": 4201
  }
}
cloud: {
  "api_url": "<https://api.prefect.cloud/api>",
  "enable_orchestration_telemetry": true,
  "max_log_size": 25000,
  "ui_url": "<https://app.prefect.cloud>"
}
deployments: {
  "default_work_pool_name": null,
  "default_docker_build_namespace": null
}
experiments: {
  "warn": true,
  "plugins": {
    "enabled": false,
    "allow": [],
    "deny": [],
    "setup_timeout_seconds": 20,
    "strict": false,
    "safe_mode": false
  }
}
flows: {
  "default_retries": 0,
  "default_retry_delay_seconds": 0
}
internal: {
  "logging_level": "WARNING"
}
logging: {
  "level": "INFO",
  "config_path": "/home/prefect/.prefect/logging.yml",
  "extra_loggers": [],
  "log_prints": false,
  "colors": true,
  "markup": false,
  "to_api": {
    "enabled": true,
    "batch_interval": 2,
    "batch_size": 4000000,
    "max_log_size": 1000000,
    "when_missing_flow": "warn"
  }
}
results: {
  "default_serializer": "pickle",
  "persist_by_default": false,
  "default_storage_block": null,
  "local_storage_path": "/home/prefect/.prefect/storage"
}
runner: {
  "process_limit": 5,
  "poll_frequency": 10,
  "heartbeat_frequency": null,
  "server": {
    "enable": false,
    "host": "localhost",
    "port": 8080,
    "log_level": "ERROR",
    "missed_polls_tolerance": 2
  }
}
server: {
  "logging_level": "INFO",
  "analytics_enabled": false,
  "metrics_enabled": true,
  "log_retryable_errors": true,
  "register_blocks_on_start": true,
  "memoize_block_auto_registration": true,
  "memo_store_path": "/home/prefect/.prefect/memo_store.toml",
  "deployment_schedule_max_scheduled_runs": 50,
  "api": {
    "auth_string": "**********",
    "host": "0.0.0.0",
    "port": 4200,
    "base_path": "/api",
    "default_limit": 200,
    "keepalive_timeout": 70,
    "csrf_protection_enabled": false,
    "csrf_token_expiration": "PT1H",
    "cors_allowed_origins": "*",
    "cors_allowed_methods": "*",
    "cors_allowed_headers": "*"
  },
  "concurrency": {
    "lease_storage": "prefect_redis.lease_storage",
    "initial_deployment_lease_duration": 300,
    "maximum_concurrency_slot_wait_seconds": 30
  },
  "database": {
    "sqlalchemy": {
      "connect_args": {
        "application_name": null,
        "statement_cache_size": null,
        "prepared_statement_cache_size": null,
        "tls": {
          "enabled": false,
          "ca_file": null,
          "cert_file": null,
          "key_file": null,
          "check_hostname": true
        }
      },
      "pool_size": 5,
      "pool_recycle": 3600,
      "pool_timeout": 30,
      "max_overflow": 10
    },
    "connection_url": "**********",
    "driver": null,
    "host": null,
    "port": null,
    "user": null,
    "name": null,
    "password": null,
    "echo": false,
    "migrate_on_start": false,
    "timeout": 30,
    "connection_timeout": 5
  },
  "deployments": {
    "concurrency_slot_wait_seconds": 30
  },
  "docket": {
    "name": "prefect-server",
    "url": "<redis://prefect-redis-replication-master.prefect.svc.cluster.local:6379/0>"
  },
  "ephemeral": {
    "enabled": true,
    "startup_timeout_seconds": 20
  },
  "events": {
    "stream_out_enabled": true,
    "related_resource_cache_ttl": "PT5M",
    "maximum_labels_per_resource": 500,
    "maximum_related_resources": 100,
    "maximum_size_bytes": 1500000,
    "expired_bucket_buffer": "PT1M",
    "proactive_granularity": "PT5S",
    "retention_period": "P7D",
    "maximum_websocket_backfill": "PT15M",
    "websocket_backfill_page_size": 250,
    "messaging_broker": "prefect_redis.messaging",
    "messaging_cache": "prefect_redis.messaging",
    "causal_ordering": "prefect_redis.ordering",
    "maximum_event_name_length": 1024
  },
  "flow_run_graph": {
    "max_nodes": 10000,
    "max_artifacts": 10000
  },
  "logs": {
    "stream_out_enabled": false,
    "stream_publishing_enabled": false
  },
  "services": {
    "cancellation_cleanup": {
      "enabled": true,
      "loop_seconds": 20
    },
    "event_persister": {
      "enabled": true,
      "batch_size": 20,
      "read_batch_size": 1,
      "flush_interval": 5,
      "batch_size_delete": 10000
    },
    "event_logger": {
      "enabled": true
    },
    "foreman": {
      "enabled": true,
      "loop_seconds": 15,
      "inactivity_heartbeat_multiple": 3,
      "fallback_heartbeat_interval_seconds": 30,
      "deployment_last_polled_timeout_seconds": 60,
      "work_queue_last_polled_timeout_seconds": 60
    },
    "late_runs": {
      "enabled": true,
      "loop_seconds": 5,
      "after_seconds": "PT15S"
    },
    "scheduler": {
      "enabled": true,
      "loop_seconds": 60,
      "deployment_batch_size": 100,
      "max_runs": 100,
      "min_runs": 3,
      "max_scheduled_time": "P100D",
      "min_scheduled_time": "PT1H",
      "insert_batch_size": 500,
      "recent_deployments_loop_seconds": 5
    },
    "pause_expirations": {
      "enabled": true,
      "loop_seconds": 5
    },
    "repossessor": {
      "enabled": true,
      "loop_seconds": 15
    },
    "task_run_recorder": {
      "enabled": true,
      "read_batch_size": 1
    },
    "triggers": {
      "enabled": true,
      "read_batch_size": 1,
      "pg_notify_reconnect_interval_seconds": 10,
      "pg_notify_heartbeat_interval_seconds": 5
    }
  },
  "tasks": {
    "tag_concurrency_slot_wait_seconds": 30,
    "max_cache_key_length": 2000,
    "scheduling": {
      "max_scheduled_queue_size": 1000,
      "max_retry_queue_size": 100,
      "pending_task_timeout": "PT0S"
    }
  }
}
m
✅ thought for 97.3 seconds
Awesome, thanks for the config snippet — this is very revealing. What jumps out - Events streaming is enabled and you’re using Redis for the event broker/ordering. Good. - The Event Logger is enabled and you even saw a “[prefect.deployment.ready]” line — that means events are being published into the system. - Critical: server.database.migrate_on_start is false. If DB migrations were never run after installing/upgrading Prefect 3.x, the events tables won’t exist and the EventPersister can’t write anything. The UI feed will stay empty. Likely root cause - Database migrations have not been applied, so the events/event_resources tables don’t exist. Your event logger can print events, but the persister won’t be able to persist them. Fix it 1) Run the DB migrations once - From any pod/container with the Prefect CLI and DB access:
Copy code
prefect server database upgrade
- Kubernetes example:
Copy code
kubectl exec -it deploy/<your-prefect-api-or-background-deploy> -- prefect server database upgrade
- Or enable auto-migrate on startup going forward:
Copy code
PREFECT_SERVER_DATABASE_MIGRATE_ON_START=true
2) Verify tables and event ingestion - Check tables and row count:
Copy code
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'public' AND table_name IN ('events', 'event_resources');

SELECT COUNT(*) FROM events;
- Post a test event:
Copy code
curl -X POST http://<api-host>:<port>/api/events \
  -H "Content-Type: application/json" \
  -d '[{
    "occurred": "'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'",
    "event": "diagnostic.test",
    "resource": {"prefect.resource.id": "diagnostic/test"}
  }]'
- Re-check:
Copy code
SELECT COUNT(*) FROM events;
- And via API:
Copy code
curl -X POST http://<api-host>:<port>/api/events/filter \
  -H "Content-Type: application/json" \
  -d '{"filter": {"events": ["*"]}, "limit": 5}'
3) Ensure the feed can stream to the UI - Your config has events.stream_out_enabled: true. If the DB now contains events but the UI feed is still blank, check the browser Network tab for a WebSocket to 
/api/events/out
. If it fails, update your ingress to allow WebSocket upgrades and long-lived connections. Example for NGINX:
Copy code
location /api/events/out {
  proxy_pass http://<prefect-api-service>;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_read_timeout 3600s;
  proxy_send_timeout 3600s;
}
Optional diagnostics - Make sure the persister is actually starting after restart:
Copy code
kubectl logs deploy/<your-prefect-api-or-background-deploy> | egrep -i "event_persister|events.services|Distributor|websocket"
- Temporarily increase verbosity to catch errors:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
Given your settings (Redis broker/ordering enabled, stream_out true, event_logger true), my money is on the missing migrations. Run the upgrade and let me know: - Whether the tables now exist - The result of the test-event POST and subsequent queries - If the UI WebSocket shows as connected in your browser If anything still looks off, please share the result of those checks and I’ll dig right in.
14 Views
Open in Slack
PreviousNext
Powered by