https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am self hosting a Prefect 3 server....
# ask-marvin
n
@Marvin I am self hosting a Prefect 3 server. I setup zombie flows detection and it works. However, using the Prefect server helm chart, I have divided the background service and added a Redis instance for event handling. It works, but I don't heartbeat event in the UI anymore. Every other event is present. Why?
m
✅ thought for 142.1 seconds
Heartbeat events are a bit special in Prefect 3, and the behavior you’re seeing is actually explainable from the server architecture. Short version: 
prefect.flow-run.heartbeat
events are primarily used by the server for liveness/zombie detection, and in a split-services self-hosted setup they may be persisted/consumed for detection without being streamed back to the UI event feed the same way you expect. If zombie detection works, the heartbeats are almost certainly being emitted and ingested; the missing piece is usually the live event stream path to the UI, not heartbeat generation itself. What I verified from Prefect 3 source: - Heartbeats are emitted as normal events: 
prefect.flow-run.heartbeat
- Zombie detection relies on those events being available server-side - Event persistence and event streaming are separate concerns - In a split deployment: - background services handle persistence/processing - API/webserver processes handle streaming events to the UI So if: - zombie detection works - other events show up in the UI - heartbeat events specifically do not then the most likely explanations are: 1. Heartbeat events are being used/persisted but not surfaced in the UI feed the same way 2. Your API/web pods are not handling the event stream identically after the Redis-backed split 3. There’s filtering or stream behavior affecting only the high-volume heartbeat event type A few concrete things to check: 1. Make sure the API/webserver pods are the ones running event stream-out If you split services, the usual pattern is: - API/web: 
prefect server start --no-services
- background services: 
prefect server services start
The UI gets live events through the API/webserver side, not the background services side. 2. Verify both deployments share the same messaging config Especially: - 
PREFECT_MESSAGING_BROKER
- Redis messaging settings - any cache/dedup settings If background services and API services are pointed differently, persistence can work while UI streaming is incomplete. 3. Check that stream-out is enabled There is a server setting controlling outbound event streaming: - 
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED
If disabled on the API/web pods, the UI won’t get live streamed events. 4. Check whether heartbeats exist in storage even though the UI doesn’t show them If they’re in the DB / being consumed for zombie detection, then this is specifically a UI-stream/display issue, not an ingestion issue. Why this happens conceptually Heartbeats are high-frequency operational events. Prefect gives them special handling for retention/liveness purposes. In practice, they’re not always as visible in the UI as “user-facing” state transition events. That’s especially noticeable once you separate event persistence from event streaming with Redis. So the answer to “Why?” is: Because heartbeat events are mainly infrastructure/liveness signals, and in a split self-hosted server architecture the path that powers zombie detection can still work even when those events are not appearing in the UI event stream. If you want, I can help you narrow this down to your exact Helm values. The most useful next step would be to compare the env/config for: - API/webserver pods - background service pods especially the 
PREFECT_MESSAGING_*
, Redis, and 
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED
settings. Useful references: - Self-hosted Prefect server docs: Self-hosted Prefect - Prefect server Helm chart repo: prefect-helm - Prefect source repo: Prefect GitHub If you paste your Helm values for the server/background split, I can help spot the likely mismatch quickly.
n
@Marvin • 
PREFECT_SERVER_EVENTS_STREAM_OUT_ENABLED
is enabled • 
PREFECT_MESSAGING_*
is correctly setup on background services and server (this is automatically handled by the helm chart) • server uses 
prefect server start --no-services
and background services uses 
prefect server services start
m
✅ thought for 182.9 seconds
Thanks — with those settings confirmed, this looks like expected behavior, not a Helm or Redis misconfiguration. Why you don’t see heartbeat events in the UI: 
prefect.flow-run.heartbeat
events are intentionally not surfaced in the UI event feed, even though they are still: - emitted by running flows - ingested by the server - persisted - used for zombie detection / automations I re-checked the Prefect 3 code path: - The flow engine emits 
prefect.flow-run.heartbeat
- The server event APIs and filters do not appear to exclude heartbeat events - The main UI repo only explicitly excludes 
prefect.log.write
- That means the heartbeat hiding is coming from the UI event components/library behavior rather than your server config So your setup is consistent with: - heartbeats exist - zombie detection works - UI feed does not show them That combination strongly suggests this is by design to avoid flooding the UI with high-volume heartbeat noise. Bottom line: If zombie detection works, your heartbeat events are almost certainly being processed correctly. Their absence in the UI is expected UX behavior, not evidence that Redis, the split background services, or 
--no-services
is broken. If you want to verify directly, query the events API for that event name instead of relying on the UI feed. Example:
Copy code
bash
curl -X POST http://<your-prefect-server>/api/events/filter \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "event": {
        "name": ["prefect.flow-run.heartbeat"]
      }
    }
  }'
If that returns heartbeat events, then everything is functioning as expected. Relevant links: - Self-hosted Prefect docs - Prefect source - Prefect Helm chart If you want, I can also help you with: - an API query that proves heartbeat events are present, or - a small script to inspect them from a self-hosted Prefect 3 server.
n
@Marvin can you provide a Python script tu query heartbeat events data?
m
✅ thought for 82.8 seconds
Absolutely — here’s a simple Python script that queries heartbeat events from your self-hosted Prefect 3 server using the Python client.
Copy code
python
import asyncio
from datetime import timedelta

from prefect.client.orchestration import get_client
from prefect.events.filters import EventFilter, EventNameFilter, EventOccurredFilter
import prefect.types._datetime


async def main():
    now = prefect.types._datetime.now("UTC")

    event_filter = EventFilter(
        event=EventNameFilter(
            name=["prefect.flow-run.heartbeat"]
        ),
        occurred=EventOccurredFilter(
            since=now - timedelta(hours=24),
            until=now,
        ),
    )

    async with get_client() as client:
        page = await client.read_events(
            filter=event_filter,
            limit=100,
        )

        print(f"Total matching events: {page.total}")
        print(f"Returned in first page: {len(page.events)}")
        print()

        for event in page.events:
            print(f"Event: {event.event}")
            print(f"Occurred: {event.occurred}")
            print(f"Resource ID: {event.resource.id}")
            print(f"Resource labels: {event.resource}")
            print(f"Related: {event.related}")
            print("-" * 80)

        # Optional: paginate through additional pages
        next_page = page
        while next_page.next_page:
            next_page = await next_page.get_next_page(client)
            print(f"Fetched another page with {len(next_page.events)} events")
            for event in next_page.events:
                print(f"{event.occurred} | {event.event} | {event.resource.id}")


if __name__ == "__main__":
    asyncio.run(main())
A few notes: - This uses your current Prefect client configuration, so make sure your environment is pointed at your self-hosted server, e.g.: - 
PREFECT_API_URL=http://<your-server>/api
- It queries for: - event name: 
prefect.flow-run.heartbeat
- last 24 hours If you want a smaller, more practical version that just prints timestamps + flow run IDs:
Copy code
python
import asyncio
from datetime import timedelta

from prefect.client.orchestration import get_client
from prefect.events.filters import EventFilter, EventNameFilter, EventOccurredFilter
import prefect.types._datetime


async def main():
    now = prefect.types._datetime.now("UTC")

    event_filter = EventFilter(
        event=EventNameFilter(name=["prefect.flow-run.heartbeat"]),
        occurred=EventOccurredFilter(
            since=now - timedelta(hours=24),
            until=now,
        ),
    )

    async with get_client() as client:
        page = await client.read_events(filter=event_filter, limit=50)

        for event in page.events:
            print(f"{event.occurred} | {event.resource.id}")


if __name__ == "__main__":
    asyncio.run(main())
If you want, I can also give you either: - a sync version - a version filtered to a specific 
flow_run_id
- a version that outputs JSON/CSV
n
@Marvin I tested it and I don't see any heartbeat event since splitting server and background services
m
✅ thought for 178.4 seconds
Thanks — that changes the diagnosis. If your direct API query returns no heartbeat events at all after splitting API and background services, then this is not just a UI filtering issue. It means the heartbeat events are not making it into event storage. Also, an important correction to my earlier reasoning: Zombie flow detection working does not prove flow-run heartbeat events are being persisted. Those are different mechanisms. What’s most likely happening In Prefect 3, there are two different heartbeat concepts that are easy to conflate: 1. Flow-run heartbeat events - event name: 
prefect.flow-run.heartbeat
- emitted by the flow engine - goes through the events pipeline - used for flow liveness/event-driven features 2. Worker heartbeats / service liveness - worker/service updates sent via API / DB-backed mechanisms - used by server services like worker/foreman-style health checks - not the same as event persistence So it is entirely possible for: - zombie/offline detection to work - while 
prefect.flow-run.heartbeat
events are missing from the event store That’s consistent with what you’re seeing. Most likely root cause in a split deployment The highest-probability issue is that the event ingestion path from flow process -> API -> broker -> persister is broken or partially broken after the split. In practical terms, one of these is usually true: 1. The split deployment is still effectively using in-memory messaging somewhere Even if Helm “sets it automatically”, one pod may still have: - 
PREFECT_MESSAGING_BROKER
defaulting to in-memory - 
PREFECT_MESSAGING_CACHE
defaulting to in-memory If API and background pods don’t share the exact same non-memory broker/cache, the API may receive events but the persister in the background pod won’t consume them. 2. The event persister service is not consuming from the same broker/topic Heartbeats are emitted into the event system, but if the 
EventPersister
in the background service is not reading the same stream/topic, they never reach the DB, so: - UI won’t show them - API event query won’t return them 3. The client-side event worker in the flow runtime is failing to send heartbeats Heartbeats are emitted asynchronously by the events worker. If that worker is failing to connect to 
/events/in
, or dropping events, you can lose heartbeat events before they ever reach the server. This can happen even while the flow itself runs normally. Why heartbeat events are especially likely to expose this Other events may still appear because: - they are less frequent - they may be emitted at state transitions - timing/retry behavior may differ Heartbeat events are periodic and rely on the whole chain continuously working. Concrete things to check next 1. Verify effective env vars in both pod types, not just Helm values Run this on both an API pod and a background-services pod:
Copy code
bash
env | grep PREFECT_MESSAGING
env | grep PREFECT_REDIS
env | grep PREFECT_SERVER_EVENTS
You want to confirm they are actually identical where relevant, especially: - 
PREFECT_MESSAGING_BROKER
- 
PREFECT_MESSAGING_CACHE
If either is still using memory anywhere, that’s a smoking gun. 2. Check background service logs for event persister activity Look for logs mentioning: - 
EventPersister
- event consumption - persisting/writing events - Redis/broker connection errors 3. Check API logs for 
/events/in
handling You want to see whether the API is receiving event websocket traffic from flow runs at all. 4. Enable debug logging for event internals On relevant components, set debug logging and reproduce a run:
Copy code
bash
PREFECT_LOG_LEVEL=DEBUG
PREFECT_DEBUG_MODE=true
If possible, also target event loggers in your deployment. *5. Confirm whether any new events are persisted after the split* Not just heartbeats. Query for a recent event like flow run state transitions.
If other events are queryable via API but heartbeat events are not, that points more toward runtime heartbeat emission specifically. If none of the expected recent events are queryable, that points to event persistence/broker split. 6. Check heartbeat emission settings There are flow heartbeat settings involved. If heartbeat frequency is disabled or altered unexpectedly in runtime env, heartbeat events may simply never be emitted. My current best read Given your newest data, I’d rank the likely causes as: 1. Messaging broker/cache mismatch across split services 2. Event persister not correctly consuming after the split 3. Flow runtime events worker not successfully sending heartbeat events 4. less likely: heartbeat frequency/config disabled in runtime If you want, I can help you narrow this down very quickly with a targeted checklist. Please paste the output of these from: - one API pod - one background-services pod - one flow-run execution environment (if accessible)
Copy code
bash
env | grep PREFECT_MESSAGING
env | grep PREFECT_REDIS
env | grep PREFECT_SERVER_EVENTS
env | grep PREFECT_API_URL
And if you have it, one snippet of background-service logs around startup would help too.
n
@Marvin Again, everything worked before this split, my configuration did not change
m
✅ thought for 14.5 seconds
That’s a very strong clue. If the configuration did not change and heartbeat events disappeared only after splitting API and background services, then the issue is almost certainly architectural, not a heartbeat setting in your flows. In other words: same config + different topology + only after split = cross-process event pipeline problem The most likely explanation is: Before the split - API server and background services were in the same process/pod - in-memory event broker/cache worked fine - heartbeat events could be published and consumed locally After the split - API and background services are now separate processes/pods - if any part of the event system is still process-local, heartbeats no longer make it from API ingestion to background persistence That can happen even if your Helm values did not change, because splitting changes whether in-memory defaults are still valid. Also, your statement strongly weakens the “heartbeat emission disabled” theory. If flows were emitting heartbeats before and only topology changed, the flow engine is probably still fine. So I’d focus on this specific conclusion: The split likely broke event persistence across service boundaries, even though the deployment “works” overall. Two important nuances: 1. “Helm chart handles it automatically” does not guarantee the effective runtime is identical The rendered env in the running pods is what matters. 2. Zombie detection still does not prove flow heartbeat event persistence It only proves some liveness mechanism still works. At this point, the most useful next step is not broad debugging — it’s proving whether the split introduced a process-local broker/cache anywhere. Please compare the actual runtime env from both pod types:
Copy code
bash
kubectl exec <api-pod> -- env | grep PREFECT_MESSAGING
kubectl exec <api-pod> -- env | grep PREFECT_REDIS

kubectl exec <background-pod> -- env | grep PREFECT_MESSAGING
kubectl exec <background-pod> -- env | grep PREFECT_REDIS
What I’d specifically look for is whether either pod still has something like: - 
prefect.server.utilities.messaging.memory
instead of Redis-backed messaging. If you want, paste those outputs and I’ll tell you immediately whether that’s the issue. One more possibility, if the env is truly identical: - the Helm split may have changed which services run where in a way that leaves event persistence incomplete for flow heartbeat events - that would be more like a Helm/chart or Prefect self-hosted limitation/bug than a user config issue If your envs look correct, I’d then treat this as a likely product bug/limitation in split self-hosted mode rather than misconfiguration. Relevant links: - Self-hosted Prefect docs - Prefect Helm chart - Prefect source If you paste: - API pod env for 
PREFECT_MESSAGING*
/ 
PREFECT_REDIS*
- background pod env for same I can help confirm whether this is config drift vs. a genuine split-services issue.
n
@Marvin Background services env variables:
Copy code
- name: HOME
  value: /home/prefect
- name: PREFECT_DEBUG_MODE
  value: "false"
- name: PREFECT_LOGGING_SERVER_LEVEL
  value: INFO
- name: PREFECT_UI_ENABLED
  value: "false"
- name: PREFECT_API_DATABASE_CONNECTION_URL
  valueFrom:
    secretKeyRef:
      key: connection-string
      name: prefect-server-secret
- name: PREFECT_API_DATABASE_MIGRATE_ON_START
  value: "false"
- name: PREFECT_MESSAGING_BROKER
  value: prefect_redis.messaging
- name: PREFECT_MESSAGING_CACHE
  value: prefect_redis.messaging
- name: PREFECT_SERVER_EVENTS_CAUSAL_ORDERING
  value: prefect_redis.ordering
- name: PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE
  value: prefect_redis.lease_storage
- name: PREFECT_REDIS_MESSAGING_HOST
  value: valkey.prefect-server.svc.cluster.local
- name: PREFECT_REDIS_MESSAGING_PORT
  value: "6379"
- name: PREFECT_REDIS_MESSAGING_DB
  value: "0"
- name: PREFECT_SERVER_ANALYTICS_ENABLED
  value: "false"
- name: PREFECT_API_SERVICES_LATE_RUNS_ENABLED
  value: "false"
- name: PREFECT_SERVER_API_KEEPALIVE_TIMEOUT
  value: "20"
Server env variables:
Copy code
- name: HOME
  value: /home/prefect
- name: PREFECT_DEBUG_MODE
  value: "false"
- name: PREFECT_LOGGING_SERVER_LEVEL
  value: INFO
- name: PREFECT_SERVER_API_BASE_PATH
  value: /api
- name: PREFECT_SERVER_API_HOST
  value: 0.0.0.0
- name: PREFECT_SERVER_API_PORT
  value: "4200"
- name: PREFECT_UI_API_URL
  value: <https://orchestrator.latitudo40.com/api>
- name: PREFECT_UI_STATIC_DIRECTORY
  value: /ui_build
- name: PREFECT_API_DATABASE_CONNECTION_URL
  valueFrom:
    secretKeyRef:
      key: connection-string
      name: prefect-server-secret
- name: PREFECT_SERVER_API_AUTH_STRING
  valueFrom:
    secretKeyRef:
      key: auth-string
      name: prefect-server-secret
- name: PREFECT_SERVER_ANALYTICS_ENABLED
  value: "false"
- name: PREFECT_API_SERVICES_LATE_RUNS_ENABLED
  value: "false"
- name: PREFECT_SERVER_API_KEEPALIVE_TIMEOUT
  value: "20"
- name: PREFECT_SERVER_UI_SHOW_PROMOTIONAL_CONTENT
  value: "false"
- name: PREFECT_SERVER_METRICS_ENABLED
  value: "true"
- name: PREFECT_API_DATABASE_MIGRATE_ON_START
  value: "false"
- name: PREFECT_MESSAGING_BROKER
  value: prefect_redis.messaging
- name: PREFECT_MESSAGING_CACHE
  value: prefect_redis.messaging
- name: PREFECT_SERVER_EVENTS_CAUSAL_ORDERING
  value: prefect_redis.ordering
- name: PREFECT_SERVER_CONCURRENCY_LEASE_STORAGE
  value: prefect_redis.lease_storage
- name: PREFECT_REDIS_MESSAGING_HOST
  value: valkey.prefect-server.svc.cluster.local
- name: PREFECT_REDIS_MESSAGING_PORT
  value: "6379"
- name: PREFECT_REDIS_MESSAGING_DB
  value: "0"
m
✅ thought for 171.1 seconds
Thanks — these envs are very helpful. Based on what you shared, your Redis messaging configuration looks correct on both the API and background-services pods. So I no longer think this is a simple 
PREFECT_MESSAGING_BROKER
/ 
PREFECT_MESSAGING_CACHE
mismatch. That leaves a narrower set of likely causes. Most likely remaining issue: background services topology, not messaging config In Prefect 3, the event persister is what takes events off the broker and writes them to the database. In a split deployment: - 
prefect server start --no-services
- serves API/UI - receives events - does not run the persister - 
prefect server services start
- should run the background services, including the event persister So if heartbeat events vanished only after the split, and your Redis settings are correct, the next most likely explanations are: 1. The background-services pod is not actually running the EventPersister correctly This is now my top suspicion. 2. Docket/service coordination is not configured for split services This is another important one. In multi-process/multi-pod service setups, Prefect often needs a shared docket backend rather than in-memory coordination. I do not see 
PREFECT_SERVER_DOCKET_URL
in the env you pasted. That matters. If it is unset, Prefect may default to in-memory docketing, which is fine in a single combined process but can break service coordination once you split API and background services. What I would check next 1. Check whether 
PREFECT_SERVER_DOCKET_URL
is set anywhere On the background-services pod:
Copy code
bash
env | grep PREFECT_SERVER_DOCKET_URL
If it’s missing, that is a very plausible cause. For split/self-hosted deployments, you generally want it backed by Redis, not memory. 2. Confirm EventPersister is actually running Please check the background service logs for startup lines mentioning 
EventPersister
. If you can, run:
Copy code
bash
kubectl logs <background-pod>
and search for: - 
EventPersister
- 
event persister
- 
services
- 
error
- 
traceback
3. Check whether recent non-heartbeat events are being persisted at all This is key to scoping the issue. If your direct events API query can find recent flow state events but not heartbeats, then this is heartbeat-specific. If no recent events are present after the split, then the persister path is broken more generally. 4. Verify the background-services process is actually healthy Make sure the pod is not restarting or partially failing. Why I think docket is worth checking now Your “everything worked before the split, config did not change” statement strongly suggests that something that was safe as a single-process default became unsafe in a multi-process topology. 
PREFECT_SERVER_DOCKET_URL
is exactly that kind of setting. My current best diagnosis Given everything you’ve shown: - Not likely: Redis messaging env mismatch - Likely: service coordination / persister startup issue after split - Most suspicious missing setting: 
PREFECT_SERVER_DOCKET_URL
So I’d check that next. If you want, send me:
Copy code
bash
env | grep PREFECT_SERVER_DOCKET_URL
from the background pod, and ideally a snippet of the background pod logs at startup, and I can help interpret whether this is the root cause. References: - Self-hosted Prefect docs - Prefect source - Prefect Helm chart
n
@Marvin I'm using chart version 2026.1.20181753, there is option to setup docket
m
✅ thought for 17.4 seconds
Got it — and that’s useful context. If you’re on Helm chart 
2026.1.20181753
and there’s no chart option to configure docket, then I would not assume this is something you missed in values.yaml. Given everything you’ve shared, my current read is: This looks more like a Prefect/Helm split-services limitation or bug than a user misconfiguration. Why I say that: - it worked before splitting - your Redis event settings look correct on both pod types - API uses 
prefect server start --no-services
- background uses 
prefect server services start
- direct event queries show no heartbeat events persisted - chart apparently does not expose docket configuration That combination makes “you forgot a required config knob” much less convincing. So there are two realistic possibilities left: 1. The chart’s split-services mode is incomplete for heartbeat event persistence Meaning the deployment topology is supported in general, but heartbeat events are falling through a gap. 2. Prefect server requires an additional shared-coordination setting for split mode, but the chart version you’re using does not expose it In that case, that’s still effectively a chart/product issue, not something obvious from your side. What I would do next Since we’ve narrowed this down quite a bit, the most useful next step is to verify whether the event persister is actually active in the background pod. Please check background pod logs around startup for anything mentioning: - 
EventPersister
- 
Starting
- 
service
- 
error
- 
traceback
If you can, this command is a good start:
Copy code
bash
kubectl logs <background-pod> --since=30m
If you want a tighter filter:
Copy code
bash
kubectl logs <background-pod> --since=30m | grep -Ei "event|persister|service|error|traceback"
Why this matters If the persister is not starting, or is erroring after startup, that would exactly explain: - events not showing up in API queries - heartbeat events disappearing after split - everything else seeming mostly fine My recommendation At this point, I’d treat this as likely worthy of a GitHub issue if logs confirm the persister isn’t functioning correctly in split mode. Before doing that, the ideal evidence bundle would be: 1. your chart version: 
2026.1.20181753
2. confirmation that combined mode worked 3. confirmation that split mode removes heartbeat events from event queries 4. the matching Redis envs you already pasted 5. background pod logs showing persister startup or failure If you want, paste the background pod startup logs here and I’ll help determine whether this is enough to conclude “likely Prefect/Helm bug” and help you draft a concise GitHub issue.
9 Views
Open in Slack
PreviousNext
Powered by