<@ULVA73B9P> I set up an automation to detect zomb...
# ask-marvin
s
@Marvin I set up an automation to detect zombies flows as proposed in the documentation. However, when I restart the container running the flow, it never switches the flow to "crashed" even though I see the heartbeats have stopped. I'm using prefect 3.4.20 in docker-compose. I scheduled the flow with
await run_deployment(...)
function. What is the correct way of doing it ?
m
thought for 649.8 seconds
Short answer: in Prefect 3.x, runs only crash on missed heartbeats if 1) the flow process is actually emitting heartbeats, and 2) you use a proactive automation that reacts to the absence of heartbeats within a window. A few common gotchas can prevent the crash action from firing. What to check and how to set it up correctly 1) Ensure heartbeats are enabled for the process that runs the flow - Set
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
to a value >= 30 seconds in the runtime environment of the flow process. - ProcessWorker: put this env var on the worker container (the flow runs as a subprocess of the worker). - DockerWorker: put this env var in the job/container environment (image or job variables). - If you only set it on your server or the client calling
run_deployment(...)
, your flow will not emit heartbeats. - You can confirm it’s working by opening the flow run’s Timeline and verifying
flow-run.heartbeat
events appear. Docs: Detect zombie flows 2) Use a proactive “absence of event” automation - Prefect does not emit a “heartbeat missed” event. If you created a reactive automation that listens for a non-existent event, it won’t trigger. - Instead, create a proactive automation that says: - For each flow run in state RUNNING - Expect
prefect.flow-run.heartbeat
every N seconds (your heartbeat frequency) - Within 2–3x that interval (e.g., 60–120s if frequency is 30s) - If no heartbeat OR terminal state event occurs, Change Flow Run State -> Crashed - Create this in the UI (recommended). CLI does not support “create”; you can manage existing automations with
prefect automations ls|inspect|pause|resume
, but creation is done via the UI or Python. 3) Make the timing tolerant enough - If your heartbeat frequency is 30s, set the automation’s “within” window to at least 90s (3x). Too tight a window can cause flakiness and false negatives. - After restarting the container, wait the full “within” window for the automation to fire. 4) Self-hosted Prefect Server specifics - Ensure your automations service is running (it is included in the default server compose). Your heartbeats appearing in the Timeline indicate events are flowing; the automation should then be able to act. 5)
run_deployment(...)
is fine - Scheduling with
await run_deployment(...)
does not affect heartbeat/crash behavior. What matters is whether the runtime process of the flow emits heartbeats. Example snippets - ProcessWorker in docker-compose
Copy code
services:
  worker:
    image: prefecthq/prefect:3.4.20-python3.11
    environment:
      PREFECT_API_URL: <http://server:4200/api>
      PREFECT_RUNNER_HEARTBEAT_FREQUENCY: "30"
      PREFECT_LOGGING_LEVEL: INFO
    command: >
      prefect worker start --pool process-pool
- DockerWorker deployment (set the env on the job)
Copy code
flow.from_source("<https://github.com/org/repo>", entrypoint="flows/my_flow.py:flow").deploy(
  name="my-deploy",
  work_pool_name="docker-pool",
  job_variables={
    "image": "myimage:tag",
    "env": {
      "PREFECT_RUNNER_HEARTBEAT_FREQUENCY": "30"
    },
  },
)
How to test quickly - Create a long-running flow (e.g., sleep for 10 minutes), start a run, verify heartbeats appear. - Restart/kill the container running the flow. - Wait > your automation window (e.g., >90–120s). The automation should change the run to Crashed. If it still doesn’t work, please share: - Where you set
PREFECT_RUNNER_HEARTBEAT_FREQUENCY
(worker vs job container) - Your worker type (ProcessWorker vs DockerWorker) - A screenshot or exact details of your automation trigger (posture, expected event, within window, filters) - Whether you’re on Prefect Cloud or self-hosted Server Links - Detect zombie flows: Docs: Detect zombie flows - Self-hosted with Docker Compose: Docs: Docker Compose
s
@Marvin When I try with a dummy flow as suggested, I see the runner is actually cancelling the flow before being stopped. Why does it cancel flows automatically while it does not with my original flow ? is it because of sub-flows ?
m
thought for 91.2 seconds
Great observation — that behavior is expected based on how the process is being stopped and how the work is structured. Why the dummy flow cancels “automatically” - Prefect’s engine traps SIGTERM/SIGINT and will attempt a graceful shutdown: it transitions the flow to Cancelling → Cancelled and cancels running tasks/subflows. Docker Compose’s restart/stop sends SIGTERM then waits a few seconds before SIGKILL, so your dummy flow gets time to cancel. - If you want a run to show up as Crashed, the process must die ungracefully (e.g., SIGKILL or a hard container crash) so it cannot report a cancellation. That’s when heartbeats stop and the proactive “absence of heartbeats” automation should mark it Crashed. Why your original flow may behave differently This usually comes down to structure and where the work actually runs: - Subflows vs child deployments: - Subflows (calling another @flow inside your flow) run in the same process. They’ll be cancelled along with the parent on SIGTERM. -
run_deployment(...)
launches entirely separate flow runs. Those child runs are independent processes/containers. Stopping the parent container does not cancel those remote runs. They’ll keep running unless you cancel them explicitly or their containers die. - Background tasks (
.delay()
and
.result_async()
): - If you’re using Prefect’s task workers with
.delay()
, those tasks are still tied to the parent run. On a graceful shutdown, Prefect will transition the parent to Cancelling and mark child task runs accordingly; workers should stop picking up work for the cancelled run. - How you stop the container: -
docker compose restart/stop
= SIGTERM → graceful cancellation path. -
docker kill -s KILL
or a crash = no time to cancel → run remains Running and, if heartbeats were enabled in that runtime, your proactive automation should mark it Crashed after the window. What to try next - To verify crash detection: kill the actual job container that’s running the flow with SIGKILL. - Example:
docker kill --signal=KILL <container-running-the-flow>
- Make sure heartbeats are enabled in the environment of the process actually running the flow: - Set
PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30
(or higher) in the job/worker container env, not just on your server or the client that calls
run_deployment(...)
. - If you want child runs to cancel with the parent: - Prefer subflows instead of
run_deployment(...)
, or - Add an automation or external cleanup step that cancels the child runs when the parent is Cancelling/Cancelled (e.g., listen for parent state change and use the API to cancel children identified by run IDs/tags). A couple of clarifying questions to make sure we’re targeting the right process: - In your “original” flow, are you fanning out with
run_deployment(...)
(separate flow runs) or subflows? - Which container are you restarting/killing — the parent launcher/worker, or the actual job container that’s running the flow? - How are you stopping it (docker compose restart/stop vs docker kill)? Once I have those details, I can suggest the exact setup to get consistent “Crashed” detection or coordinated cancellation across your runs.