<@ULVA73B9P> We're using cloud run v2 push work po...
# ask-marvin
m
@Marvin We're using cloud run v2 push work pools and have the
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
environment variable set in the work pool template. I'm however not seeing any heartbeat events in the Prefect UI. Is there any known limitation with cloud run and heartbeat events?
m
thought for 346.4 seconds
Short answer: there isn’t a Cloud Run–specific limitation here, but in Prefect 3.x runner heartbeats are not surfaced as user-visible “events” in the UI. Setting
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
works, but you won’t see periodic heartbeat entries in the Events timeline; the heartbeat is used internally by the server to detect inactivity. A few clarifications and checks: - Push pools vs pull workers: With Cloud Run v2 push work pools there is no long-lived worker process, so there are no worker heartbeats. The only heartbeat is from the flow’s runner process while your job container is running. - Minimum interval: The setting must be >= 30 seconds. If your flow runs are shorter than the heartbeat interval, you won’t get any heartbeats during those runs. - Confirm the setting is applied inside the job container. An easy way is to log it at flow start:
Copy code
from prefect import flow, get_run_logger
  from prefect.settings import get_current_settings

  @flow
  def demo():
      logger = get_run_logger()
      <http://logger.info|logger.info>(f"Runner heartbeat frequency: {get_current_settings().runner.heartbeat_frequency}")

  if __name__ == "__main__":
      demo()
- Networking: Ensure the job can egress to the Prefect API; heartbeats are outbound HTTP calls. If your goal is to alert on “no heartbeat” conditions, let me know what you want to drive (e.g., an automation or notification) and I can suggest the best approach in 3.x for your setup. Questions to help me tailor further: - Which Prefect version are you on? - Where in the UI were you expecting to see the heartbeat (Flow Run event stream, Work Pool view, etc.)? - Roughly how long do your Cloud Run jobs typically run?
m
@Marvin The job is able to access http endpoints, sync with rclone etc. so egress shouldn't be an issue. I've also confirmed that the prefect version and that the heartbeat setting is as required, i.e when doing the below. I'm expecting to see the heartbeats in the "Events" section, i've been able to deploy a flow locally that emittted events and those did show up in the UI, The job is running for several minutes, that is not the issue
Copy code
Part of my code:    
logger.info(f"Prefect version: {prefect.__version__}")
    logger.info(
        f"Runner heartbeat frequency: {get_current_settings().runner.heartbeat_frequency}"
    )

Result: 
2025-12-08 19:12:41.559 CET 18:12:41.558 | INFO | prefect - Prefect version: 3.5.0
2025-12-08 19:12:41.560 CET 18:12:41.560 | INFO | prefect - Runner heartbeat frequency: 30
m
thought for 127.1 seconds
Thanks for the extra detail — that helps. What you’re seeing is expected behavior in Prefect 3.5.0: runner heartbeats are not emitted as user-visible “events” in the UI. The
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
you set is being honored (as your logs confirm), but those heartbeats update server-side metadata used for inactivity detection rather than producing entries in the Events timeline. This isn’t specific to Cloud Run push pools. Ways to verify during a run: - Check the flow run’s last heartbeat on the API while the job is running. For example, from inside your flow:
Copy code
from prefect import flow, get_client, get_run_logger
  from prefect.runtime import flow_run
  import asyncio

  @flow
  async def show_last_heartbeat():
      logger = get_run_logger()
      async with get_client() as client:
          fr = await client.read_flow_run(flow_run.id)
          <http://logger.info|logger.info>(f"last_heartbeat_time: {fr.last_heartbeat_time}")
          # wait and check again
          await asyncio.sleep(35)
          fr = await client.read_flow_run(flow_run.id)
          <http://logger.info|logger.info>(f"last_heartbeat_time (after): {fr.last_heartbeat_time}")
You should see
last_heartbeat_time
advance while the run is active. If your goal is to have a visible signal in the Events pane: - Emit your own “heartbeat” events from the flow on a timer if you want a timeline trail:
Copy code
from prefect import flow
  from prefect.events import emit_event
  import asyncio

  @flow
  async def emit_visible_heartbeats():
      for _ in range(3):
          emit_event(event="custom.heartbeat")
          await asyncio.sleep(30)
- Or set up an Automation that alerts when a run goes inactive/late (i.e., when heartbeats stop). That’s generally more actionable than surfacing every heartbeat. To recap: - There’s no Cloud Run limitation here. - Worker heartbeats don’t apply to push pools. - Runner heartbeats in 3.x don’t appear in the Events timeline by design. If you’d like, I can open a feature request to surface runner heartbeat pings (or a summarized cadence) in the Events UI — just say the word.
m
@Marvin I don't think you're correct in the fact that heartbeat events don't show up in the UI. If I run the below code locally with heartbeats set to 30 seconds I see the heartbeat events in the prefect UI.
Copy code
@flow
def child_flow() -> None:
    logger = get_logger()
    <http://logger.info|logger.info>("Starting child flow")
    while True:
        <http://logger.info|logger.info>("Child flow sleeping 10 seconds")
        time.sleep(10)


@flow
def parent_flow() -> None:
    logger = get_logger()
    <http://logger.info|logger.info>("Starting parent flow")
    child_flow()


if __name__ == "__main__":
    parent_flow.serve(name="testdeployment")
What we ultimately want to achieve is to kill zombie flows that keep running when the infrastructure crashes. We have the below automation to do this, but it does not seem to be working. My suspicion as to why it's not working is that there is somehow something different about the heartbeat settings in a cloud run push-based work pool compared to a long-running runner that polls for work. Do you know of any other ways we can achieve the action of killing zombie flows? I would be happy with for example saying that we kill any flow that has been running for more than 24 hours and mark it as crashed, but open to your suggestions too. And please let me know if there's something wrong with my automation, or if you agree that the heartbeats somehow seem to be the issue Automation JSON:
Copy code
{
  "type": "event",
  "match": {
    "prefect.resource.id": [
      "prefect.flow-run.*"
    ]
  },
  "match_related": {},
  "after": [
    "prefect.flow-run.heartbeat"
  ],
  "expect": [
    "prefect.flow-run.heartbeat",
    "prefect.flow-run.Failed",
    "prefect.flow-run.Crashed",
    "prefect.flow-run.Cancelled",
    "prefect.flow-run.Completed"
  ],
  "for_each": [
    "prefect.resource.id"
  ],
  "posture": "Proactive",
  "threshold": 1,
  "within": 90
}
m
thought for 638.5 seconds
You’re right — thanks for calling that out. In Prefect 3.x, flow-run heartbeats do show up in the UI event feed when runner heartbeats are enabled. My earlier statement was incorrect — apologies for the confusion. A few key points to align on: - Flow-run heartbeats are events (
prefect.flow-run.heartbeat
) emitted by the runner inside your flow process. They appear in the Events feed and can drive automations. - Worker heartbeats are different; they’re API pings and do not appear as events. - In push-based serverless (e.g., Cloud Run), there’s no long-running worker, but your flow still runs inside a container with a runner. If you set
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
in that container, Prefect will emit flow-run heartbeat events from there too. If that env var isn’t set in Cloud Run, you won’t see heartbeats — which would explain your automation not firing. Why your current automation likely isn’t firing - Your trigger has
after: ["prefect.flow-run.heartbeat"]
. That means the automation only starts its monitoring window after a heartbeat is seen. - If your Cloud Run job isn’t emitting heartbeats because
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
isn’t set in that environment, the “after” gate never opens and the automation never fires. - Fix: open the monitoring window on
Running
and “expect” heartbeats or a terminal event. Try this for zombie detection (missing heartbeats)
Copy code
{
  "name": "Crash zombie flows (missing heartbeats)",
  "trigger": {
    "type": "event",
    "match": {
      "prefect.resource.id": [
        "prefect.flow-run.*"
      ]
    },
    "match_related": {},
    "for_each": [
      "prefect.resource.id"
    ],
    "after": [
      "prefect.flow-run.Running"
    ],
    "expect": [
      "prefect.flow-run.heartbeat",
      "prefect.flow-run.Completed",
      "prefect.flow-run.Failed",
      "prefect.flow-run.Cancelled",
      "prefect.flow-run.Crashed"
    ],
    "posture": "Proactive",
    "threshold": 1,
    "within": 90
  },
  "actions": [
    {
      "type": "change-flow-run-state",
      "state": "CRASHED",
      "message": "No heartbeat or terminal state within 90s while Running."
    }
  ]
}
Hard cap runs at 24 hours If you want to mark any run that stays Running for >24h as crashed, use:
Copy code
{
  "name": "Crash runs older than 24h",
  "trigger": {
    "type": "event",
    "match": {
      "prefect.resource.id": [
        "prefect.flow-run.*"
      ]
    },
    "match_related": {},
    "for_each": [
      "prefect.resource.id"
    ],
    "after": [
      "prefect.flow-run.Running"
    ],
    "expect": [
      "prefect.flow-run.Completed",
      "prefect.flow-run.Failed",
      "prefect.flow-run.Cancelled",
      "prefect.flow-run.Crashed"
    ],
    "posture": "Proactive",
    "threshold": 1,
    "within": 86400
  },
  "actions": [
    {
      "type": "change-flow-run-state",
      "state": "CRASHED",
      "message": "Run exceeded 24-hour limit."
    }
  ]
}
Also consider flow-level timeout You can set a hard cap at the flow layer to fail the run after a duration:
Copy code
@flow(timeout_seconds=86400)
def my_flow(...):
    ...
This is a good complement to the automation approach. Make sure heartbeats are enabled in Cloud Run For Cloud Run push-based pools, set the env var in the deployment’s job variables so the runner emits heartbeats:
Copy code
deployments:
- name: my-deployment
  work_pool:
    name: my-cloud-run-pool
    job_variables:
      env:
        PREFECT_RUNNER_HEARTBEAT_FREQUENCY: "30"
Important caveat about “killing” - Marking as CRASHED updates Prefect state. If a container is still alive, Prefect can’t forcibly terminate it unless there’s a cooperating process to receive the cancel/crash signal. - In “zombie” scenarios caused by infra crashes, the container is usually gone — so state changes are the right remediation. - For extra safety in Cloud Run, consider provider-level timeouts on jobs/services to ensure nothing runs past your maximum duration. If you can share: - Your Prefect version - Whether
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
is set in your Cloud Run job
- The action you attached to your automation (your JSON snippet didn’t include an action) …I can double-check specifics. Helpful refs: - Event triggers and Posture (Proactive vs Reactive) - Settings reference: PREFECT_RUNNER_HEARTBEAT_FREQUENCY - Serverless compute (Cloud Run) + job_variables - Flow timeout_seconds