Geoff Irons
11/16/2025, 11:15 PMprefect worker start and I added the --with-healthcheck flag. However, I can't see how to add a healthcheck to the task/flow runner itself.
For reference, I had an issue where a failing/blocking async task was causing a complete system hang - the task runner would become completely unresponsive, be unable to be cancelled etc.Marvin
11/16/2025, 11:15 PMMarvin
11/16/2025, 11:19 PM--with-healthcheck flag only exists for workers. For individual flow runs, Prefect 3.x relies on heartbeats and state transitions for liveness detection.
What you can do instead
- Detect and recover from hung runs (recommended)
- Enable flow runner heartbeats and use a Prefect Automation to detect “zombie” runs and take action (crash/cancel/notify).
- Set an env var on your flow runtime: PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30 (minimum 30s; default is disabled)
- Create the “Detect zombie flows” automation in the UI and choose the action you want (e.g., Change flow run state to Crashed and Send notification).
- Docs: Detect zombie flows, settings ref: Runner heartbeat setting
- Prevent blocking async tasks from hanging your run
- Add timeouts:
- Per task: @task(timeout_seconds=...)
- Per flow: @flow(timeout_seconds=...)
- For async tasks, wrap long sections with timeout_async(...) to ensure cooperative cancellation:
from prefect import flow, task
from prefect.utilities.timeout import timeout_async
@task(timeout_seconds=300)
async def do_work():
async with timeout_async(300):
# awaited async work here
...
- Isolate suspect code in separate processes to hard‑kill if it wedges the interpreter:
from prefect import flow, task
from prefect.task_runners import ProcessPoolTaskRunner
@task(timeout_seconds=120)
def cpu_or_blocking_io():
# blocking library call here
...
@flow(task_runner=ProcessPoolTaskRunner())
def my_flow():
fut = cpu_or_blocking_io.submit()
return fut.result()
Using a ProcessPoolTaskRunner helps when third‑party C extensions or blocking calls ignore/circumvent async cancellation.
- If you must expose a container health endpoint
- There’s no Prefect‑native endpoint for flow runs. If you’re on Kubernetes/ECS and need a liveness/readiness probe, use one of these patterns:
- Sidecar that queries the Prefect API for the flow run’s last heartbeat and returns 200/503 accordingly. This is robust even when your flow process is hung.
- A tiny HTTP server started alongside your flow that returns 200 only if a heartbeat file/timestamp is being updated. Note: if the interpreter is fully wedged, that server won’t respond either—sidecar is safer.
- Worker healthcheck docs for reference: Worker healthchecks
Why cancellation felt unresponsive
- In async Python, cancellation is cooperative—code has to hit an await/cancellation point. A blocking or CPU‑bound call in an async task can starve the loop and prevent cancellation from being delivered. Timeouts, process isolation, or moving blocking calls to threads/processes mitigate this.
A few clarifying questions so I can tailor concrete examples:
- Which AWS runtime are you using for flow runs (ECS, EKS, EC2/Docker, Lambda)?
- Are you on Prefect Cloud or OSS?
- Would you prefer an example of a Kubernetes/ECS health probe that checks Prefect heartbeats via a sidecar?