That usually means the flow process is still alive, but the path that emits heartbeat events back to the Prefect API has stalled or become disconnected. A few important points for Prefect 3.x: * Heartbeats are sent by the flow run process itself, not by the worker pod * Logs can keep appearing even if heartbeats stop, because stdout/log shipping and heartbeat delivery are separate paths * A healthy worker/background-services pod does not guarantee an individual running flow can still send heartbeats Since you’re on Prefect 3.4.9, one especially important caveat: In many 3.x versions around yours, flow heartbeats were not enabled by default unless

PREFECT_FLOWS_HEARTBEAT_FREQUENCY

was set explicitly. So if these “heartbeat events” were previously present and then suddenly stopped, I’d first check whether something changed in the runtime env, image, or network path. Most likely causes in Kubernetes 1. Network/WebSocket issue from the running flow pod to the Prefect API * egress policy changed * DNS issue * proxy / service mesh / LB interfering with long-lived connections 2. The flow’s internal events worker stalled * heartbeats are emitted asynchronously * logs can continue while event delivery stops 3. Resource pressure in the pod/node * heavy CPU starvation * memory pressure * thread/event loop starvation without full pod crash 4. Version/config drift *

PREFECT_FLOWS_HEARTBEAT_FREQUENCY

unset/overridden * image or environment changed between runs What I’d check first 1. Confirm heartbeat frequency is actually configured inside the flow pod Run in the same container image/runtime:

python -c "from prefect.settings import get_current_settings; print(get_current_settings().flows.heartbeat_frequency)"

If that prints

None

, heartbeats are disabled. 2. Turn on debug logs for a repro Add:

PREFECT_LOGGING_LEVEL=DEBUG

Then look for messages about heartbeat startup/failure in the flow run logs. 3. Verify connectivity from the flow pod, not the worker pod From the still-running flow pod:

python -c "
   import socket
   print(socket.gethostbyname('api.prefect.cloud'))
   "

and if using Cloud:

python -c "
   import asyncio, httpx
   async def main():
       async with httpx.AsyncClient() as client:
           r = await client.get('<https://api.prefect.cloud/api/health>', timeout=10)
           print(r.status_code, r.text[:200])
   asyncio.run(main())
   "

4. Check for K8s network/policy changes * NetworkPolicies * service mesh sidecars * egress restrictions * proxy/LB idle timeout behavior 5. Check pod/node resource events

kubectl describe pod <flow-pod-name>
   kubectl get events -n <namespace> --sort-by=.lastTimestamp

Look for throttling, OOM pressure, eviction warnings, node issues. 6. Compare env between a good run and bad run Especially: *

PREFECT_API_URL

*

PREFECT_API_KEY

*

PREFECT_FLOWS_HEARTBEAT_FREQUENCY

* any proxy vars like

HTTP_PROXY

,

HTTPS_PROXY

,

NO_PROXY

Important nuance If the run remains in

Running

after heartbeats stop, that is a known class of behavior: Prefect generally does not assume “worker healthy” == “flow healthy”, so zombie/unresponsive detection is typically handled via automations. Docs: * Detect and respond to zombie flows Related issues: * When a process worker goes down, mark running tasks/flows as Crashed * Flow still in Running state when agent is down Given your description, my best guess is: the flow pod lost the ability to deliver events to the API while continuing to execute normally. If you want, I can help you narrow this down quickly if you share:

* whether you’re on Prefect Cloud or self-hosted server * whether the missing heartbeat is visible in UI/automations/events, or you saw a specific log/error * how the flow is launched on K8s (Kubernetes worker, Helm worker, custom job template, etc.) * whether

PREFECT_FLOWS_HEARTBEAT_FREQUENCY

is set in the run pod

Got it — if you’re on 3.6.12 and other flows still heartbeat normally, then this is much less likely to be a global config issue and more likely to be a run-specific stall in that flow process. The key clue is: one flow run stopped heartbeating, but the pod kept running and logs kept printing That strongly suggests one of these: 1. the flow run’s heartbeat thread died or got stuck 2. the flow run’s event delivery path got wedged while normal logging kept working 3. that specific flow entered a code path that interfered with Prefect’s background services Because it’s just one flow, I would focus on what is unique about that flow rather than the cluster. Most plausible run-specific causes * Long CPU-bound / native-code section * If that flow spends a long time inside a C extension / tight compute / subprocess orchestration, the heartbeat machinery can behave differently from normal Python logging paths. * Especially if the flow does heavy pandas/numpy/pyarrow/db-driver work, or long blocking calls. * Threading / multiprocessing / forking inside the flow * If that flow launches child processes, uses

multiprocessing

, or forks after Prefect services are initialized, it can create odd background-service behavior. * This is one of the biggest “only one flow” differentiators. * A local network stall in that pod * Not a full pod/network outage, just the event/WebSocket connection for that process getting stuck. * stdout logs still work because they don’t depend on the Prefect events client. * Very high event volume or internal queue stall * If that flow emits unusually many events/logs/task transitions compared with others, its event worker may be the one that jams. What I’d check next, in order 1. Compare this flow to healthy ones Ask: * does this flow use

multiprocessing

,

subprocess

, Dask/Ray/Spark, joblib, or custom threading? * does it do long blocking DB/API calls? * does it emit way more logs/tasks/events than other flows? * did it recently change code, dependencies, or image? If yes to any of those, that’s where I’d look first. 2. Inspect the flow pod while it is in the bad state If you can exec into the still-running pod:

python - <<'PY'
import threading
for t in threading.enumerate():
    print(t.name, t.daemon, t.is_alive())
PY

This won’t label Prefect internals perfectly, but if the heartbeat/background threads are gone entirely, that’s useful evidence. 3. Enable DEBUG for that flow and look for heartbeat-related messages Set:

PREFECT_LOGGING_LEVEL=DEBUG

Then look for messages around: * heartbeat thread starting * failures emitting events * reconnect attempts * queue full / dropped events 4. Check whether the flow uses fork/spawn behavior If the flow uses multiprocessing, try to confirm whether the bad behavior correlates with entering that section of code. 5. Check whether task state updates also stop, or only heartbeat events This is important: * if all Prefect state/event traffic from that run stops, the whole client/event path likely stalled * if only heartbeat events stop but task state updates continue, that points more narrowly at heartbeat-specific machinery 6. See whether the stall happens at the same logical stage each time For example: * always during one task * always after a subprocess launch * always after a DB call * always after N hours That pattern is often the fastest way to isolate the root cause. My current best hypothesis Since: * same cluster * same Prefect version * heartbeats work for other flows * pod remains alive * logs continue …I’d currently suspect something specific in that flow is wedging Prefect’s event/heartbeat background service, most likely around: * multiprocessing/forking * long-running blocking/native code * a network/client stall that affects only that process Two concrete questions that would help narrow it down fast

1. Does this flow use multiprocessing / subprocesses / executors / Spark / Dask / Ray / joblib? 2. After heartbeat stops, do task state changes still appear in the UI, or does all run-state activity stop until the flow finishes? If you want, paste the rough structure of that flow — especially any concurrency/subprocess parts — and I can help identify likely heartbeat-breaking sections.