Syméon del Marmol
10/01/2025, 11:09 AMawait run_deployment(...) function.
What is the correct way of doing it ?Marvin
10/01/2025, 11:09 AMMarvin
10/01/2025, 11:19 AMPREFECT_RUNNER_HEARTBEAT_FREQUENCY to a value >= 30 seconds in the runtime environment of the flow process.
- ProcessWorker: put this env var on the worker container (the flow runs as a subprocess of the worker).
- DockerWorker: put this env var in the job/container environment (image or job variables).
- If you only set it on your server or the client calling run_deployment(...), your flow will not emit heartbeats.
- You can confirm it’s working by opening the flow run’s Timeline and verifying flow-run.heartbeat events appear.
Docs: Detect zombie flows
2) Use a proactive “absence of event” automation
- Prefect does not emit a “heartbeat missed” event. If you created a reactive automation that listens for a non-existent event, it won’t trigger.
- Instead, create a proactive automation that says:
- For each flow run in state RUNNING
- Expect prefect.flow-run.heartbeat every N seconds (your heartbeat frequency)
- Within 2–3x that interval (e.g., 60–120s if frequency is 30s)
- If no heartbeat OR terminal state event occurs, Change Flow Run State -> Crashed
- Create this in the UI (recommended). CLI does not support “create”; you can manage existing automations with prefect automations ls|inspect|pause|resume, but creation is done via the UI or Python.
3) Make the timing tolerant enough
- If your heartbeat frequency is 30s, set the automation’s “within” window to at least 90s (3x). Too tight a window can cause flakiness and false negatives.
- After restarting the container, wait the full “within” window for the automation to fire.
4) Self-hosted Prefect Server specifics
- Ensure your automations service is running (it is included in the default server compose). Your heartbeats appearing in the Timeline indicate events are flowing; the automation should then be able to act.
5) run_deployment(...) is fine
- Scheduling with await run_deployment(...) does not affect heartbeat/crash behavior. What matters is whether the runtime process of the flow emits heartbeats.
Example snippets
- ProcessWorker in docker-compose
services:
worker:
image: prefecthq/prefect:3.4.20-python3.11
environment:
PREFECT_API_URL: <http://server:4200/api>
PREFECT_RUNNER_HEARTBEAT_FREQUENCY: "30"
PREFECT_LOGGING_LEVEL: INFO
command: >
prefect worker start --pool process-pool
- DockerWorker deployment (set the env on the job)
flow.from_source("<https://github.com/org/repo>", entrypoint="flows/my_flow.py:flow").deploy(
name="my-deploy",
work_pool_name="docker-pool",
job_variables={
"image": "myimage:tag",
"env": {
"PREFECT_RUNNER_HEARTBEAT_FREQUENCY": "30"
},
},
)
How to test quickly
- Create a long-running flow (e.g., sleep for 10 minutes), start a run, verify heartbeats appear.
- Restart/kill the container running the flow.
- Wait > your automation window (e.g., >90–120s). The automation should change the run to Crashed.
If it still doesn’t work, please share:
- Where you set PREFECT_RUNNER_HEARTBEAT_FREQUENCY (worker vs job container)
- Your worker type (ProcessWorker vs DockerWorker)
- A screenshot or exact details of your automation trigger (posture, expected event, within window, filters)
- Whether you’re on Prefect Cloud or self-hosted Server
Links
- Detect zombie flows: Docs: Detect zombie flows
- Self-hosted with Docker Compose: Docs: Docker ComposeSyméon del Marmol
10/01/2025, 12:03 PMMarvin
10/01/2025, 12:03 PMMarvin
10/01/2025, 12:04 PMrun_deployment(...) launches entirely separate flow runs. Those child runs are independent processes/containers. Stopping the parent container does not cancel those remote runs. They’ll keep running unless you cancel them explicitly or their containers die.
- Background tasks (.delay() and .result_async()):
- If you’re using Prefect’s task workers with .delay(), those tasks are still tied to the parent run. On a graceful shutdown, Prefect will transition the parent to Cancelling and mark child task runs accordingly; workers should stop picking up work for the cancelled run.
- How you stop the container:
- docker compose restart/stop = SIGTERM → graceful cancellation path.
- docker kill -s KILL or a crash = no time to cancel → run remains Running and, if heartbeats were enabled in that runtime, your proactive automation should mark it Crashed after the window.
What to try next
- To verify crash detection: kill the actual job container that’s running the flow with SIGKILL.
- Example: docker kill --signal=KILL <container-running-the-flow>
- Make sure heartbeats are enabled in the environment of the process actually running the flow:
- Set PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30 (or higher) in the job/worker container env, not just on your server or the client that calls run_deployment(...).
- If you want child runs to cancel with the parent:
- Prefer subflows instead of run_deployment(...), or
- Add an automation or external cleanup step that cancels the child runs when the parent is Cancelling/Cancelled (e.g., listen for parent state change and use the API to cancel children identified by run IDs/tags).
A couple of clarifying questions to make sure we’re targeting the right process:
- In your “original” flow, are you fanning out with run_deployment(...) (separate flow runs) or subflows?
- Which container are you restarting/killing — the parent launcher/worker, or the actual job container that’s running the flow?
- How are you stopping it (docker compose restart/stop vs docker kill)?
Once I have those details, I can suggest the exact setup to get consistent “Crashed” detection or coordinated cancellation across your runs.