Mathias
12/08/2025, 4:53 PMPREFECT_RUNNER_HEARTBEAT_FREQUENCY environment variable set in the work pool template. I'm however not seeing any heartbeat events in the Prefect UI. Is there any known limitation with cloud run and heartbeat events?Marvin
12/08/2025, 4:53 PMMarvin
12/08/2025, 4:59 PMPREFECT_RUNNER_HEARTBEAT_FREQUENCY works, but you won’t see periodic heartbeat entries in the Events timeline; the heartbeat is used internally by the server to detect inactivity.
A few clarifications and checks:
- Push pools vs pull workers: With Cloud Run v2 push work pools there is no long-lived worker process, so there are no worker heartbeats. The only heartbeat is from the flow’s runner process while your job container is running.
- Minimum interval: The setting must be >= 30 seconds. If your flow runs are shorter than the heartbeat interval, you won’t get any heartbeats during those runs.
- Confirm the setting is applied inside the job container. An easy way is to log it at flow start:
from prefect import flow, get_run_logger
from prefect.settings import get_current_settings
@flow
def demo():
logger = get_run_logger()
<http://logger.info|logger.info>(f"Runner heartbeat frequency: {get_current_settings().runner.heartbeat_frequency}")
if __name__ == "__main__":
demo()
- Networking: Ensure the job can egress to the Prefect API; heartbeats are outbound HTTP calls.
If your goal is to alert on “no heartbeat” conditions, let me know what you want to drive (e.g., an automation or notification) and I can suggest the best approach in 3.x for your setup.
Questions to help me tailor further:
- Which Prefect version are you on?
- Where in the UI were you expecting to see the heartbeat (Flow Run event stream, Work Pool view, etc.)?
- Roughly how long do your Cloud Run jobs typically run?Mathias
12/08/2025, 6:16 PMPart of my code:
logger.info(f"Prefect version: {prefect.__version__}")
logger.info(
f"Runner heartbeat frequency: {get_current_settings().runner.heartbeat_frequency}"
)
Result:
2025-12-08 19:12:41.559 CET 18:12:41.558 | INFO | prefect - Prefect version: 3.5.0
2025-12-08 19:12:41.560 CET 18:12:41.560 | INFO | prefect - Runner heartbeat frequency: 30Marvin
12/08/2025, 6:18 PMMarvin
12/08/2025, 6:20 PMPREFECT_RUNNER_HEARTBEAT_FREQUENCY you set is being honored (as your logs confirm), but those heartbeats update server-side metadata used for inactivity detection rather than producing entries in the Events timeline. This isn’t specific to Cloud Run push pools.
Ways to verify during a run:
- Check the flow run’s last heartbeat on the API while the job is running. For example, from inside your flow:
from prefect import flow, get_client, get_run_logger
from prefect.runtime import flow_run
import asyncio
@flow
async def show_last_heartbeat():
logger = get_run_logger()
async with get_client() as client:
fr = await client.read_flow_run(flow_run.id)
<http://logger.info|logger.info>(f"last_heartbeat_time: {fr.last_heartbeat_time}")
# wait and check again
await asyncio.sleep(35)
fr = await client.read_flow_run(flow_run.id)
<http://logger.info|logger.info>(f"last_heartbeat_time (after): {fr.last_heartbeat_time}")
You should see last_heartbeat_time advance while the run is active.
If your goal is to have a visible signal in the Events pane:
- Emit your own “heartbeat” events from the flow on a timer if you want a timeline trail:
from prefect import flow
from prefect.events import emit_event
import asyncio
@flow
async def emit_visible_heartbeats():
for _ in range(3):
emit_event(event="custom.heartbeat")
await asyncio.sleep(30)
- Or set up an Automation that alerts when a run goes inactive/late (i.e., when heartbeats stop). That’s generally more actionable than surfacing every heartbeat.
To recap:
- There’s no Cloud Run limitation here.
- Worker heartbeats don’t apply to push pools.
- Runner heartbeats in 3.x don’t appear in the Events timeline by design.
If you’d like, I can open a feature request to surface runner heartbeat pings (or a summarized cadence) in the Events UI — just say the word.Mathias
12/09/2025, 8:04 AM@flow
def child_flow() -> None:
logger = get_logger()
<http://logger.info|logger.info>("Starting child flow")
while True:
<http://logger.info|logger.info>("Child flow sleeping 10 seconds")
time.sleep(10)
@flow
def parent_flow() -> None:
logger = get_logger()
<http://logger.info|logger.info>("Starting parent flow")
child_flow()
if __name__ == "__main__":
parent_flow.serve(name="testdeployment")
What we ultimately want to achieve is to kill zombie flows that keep running when the infrastructure crashes. We have the below automation to do this, but it does not seem to be working. My suspicion as to why it's not working is that there is somehow something different about the heartbeat settings in a cloud run push-based work pool compared to a long-running runner that polls for work.
Do you know of any other ways we can achieve the action of killing zombie flows? I would be happy with for example saying that we kill any flow that has been running for more than 24 hours and mark it as crashed, but open to your suggestions too. And please let me know if there's something wrong with my automation, or if you agree that the heartbeats somehow seem to be the issue
Automation JSON:
{
"type": "event",
"match": {
"prefect.resource.id": [
"prefect.flow-run.*"
]
},
"match_related": {},
"after": [
"prefect.flow-run.heartbeat"
],
"expect": [
"prefect.flow-run.heartbeat",
"prefect.flow-run.Failed",
"prefect.flow-run.Crashed",
"prefect.flow-run.Cancelled",
"prefect.flow-run.Completed"
],
"for_each": [
"prefect.resource.id"
],
"posture": "Proactive",
"threshold": 1,
"within": 90
}Marvin
12/09/2025, 8:04 AMMarvin
12/09/2025, 8:15 AMprefect.flow-run.heartbeat) emitted by the runner inside your flow process. They appear in the Events feed and can drive automations.
- Worker heartbeats are different; they’re API pings and do not appear as events.
- In push-based serverless (e.g., Cloud Run), there’s no long-running worker, but your flow still runs inside a container with a runner. If you set PREFECT_RUNNER_HEARTBEAT_FREQUENCY in that container, Prefect will emit flow-run heartbeat events from there too. If that env var isn’t set in Cloud Run, you won’t see heartbeats — which would explain your automation not firing.
Why your current automation likely isn’t firing
- Your trigger has after: ["prefect.flow-run.heartbeat"]. That means the automation only starts its monitoring window after a heartbeat is seen.
- If your Cloud Run job isn’t emitting heartbeats because PREFECT_RUNNER_HEARTBEAT_FREQUENCY isn’t set in that environment, the “after” gate never opens and the automation never fires.
- Fix: open the monitoring window on Running and “expect” heartbeats or a terminal event.
Try this for zombie detection (missing heartbeats)
{
"name": "Crash zombie flows (missing heartbeats)",
"trigger": {
"type": "event",
"match": {
"prefect.resource.id": [
"prefect.flow-run.*"
]
},
"match_related": {},
"for_each": [
"prefect.resource.id"
],
"after": [
"prefect.flow-run.Running"
],
"expect": [
"prefect.flow-run.heartbeat",
"prefect.flow-run.Completed",
"prefect.flow-run.Failed",
"prefect.flow-run.Cancelled",
"prefect.flow-run.Crashed"
],
"posture": "Proactive",
"threshold": 1,
"within": 90
},
"actions": [
{
"type": "change-flow-run-state",
"state": "CRASHED",
"message": "No heartbeat or terminal state within 90s while Running."
}
]
}
Hard cap runs at 24 hours
If you want to mark any run that stays Running for >24h as crashed, use:
{
"name": "Crash runs older than 24h",
"trigger": {
"type": "event",
"match": {
"prefect.resource.id": [
"prefect.flow-run.*"
]
},
"match_related": {},
"for_each": [
"prefect.resource.id"
],
"after": [
"prefect.flow-run.Running"
],
"expect": [
"prefect.flow-run.Completed",
"prefect.flow-run.Failed",
"prefect.flow-run.Cancelled",
"prefect.flow-run.Crashed"
],
"posture": "Proactive",
"threshold": 1,
"within": 86400
},
"actions": [
{
"type": "change-flow-run-state",
"state": "CRASHED",
"message": "Run exceeded 24-hour limit."
}
]
}
Also consider flow-level timeout
You can set a hard cap at the flow layer to fail the run after a duration:
@flow(timeout_seconds=86400)
def my_flow(...):
...
This is a good complement to the automation approach.
Make sure heartbeats are enabled in Cloud Run
For Cloud Run push-based pools, set the env var in the deployment’s job variables so the runner emits heartbeats:
deployments:
- name: my-deployment
work_pool:
name: my-cloud-run-pool
job_variables:
env:
PREFECT_RUNNER_HEARTBEAT_FREQUENCY: "30"
Important caveat about “killing”
- Marking as CRASHED updates Prefect state. If a container is still alive, Prefect can’t forcibly terminate it unless there’s a cooperating process to receive the cancel/crash signal.
- In “zombie” scenarios caused by infra crashes, the container is usually gone — so state changes are the right remediation.
- For extra safety in Cloud Run, consider provider-level timeouts on jobs/services to ensure nothing runs past your maximum duration.
If you can share:
- Your Prefect version
- Whether PREFECT_RUNNER_HEARTBEAT_FREQUENCY is set in your Cloud Run jobMarvin
12/09/2025, 8:15 AM