<@ULVA73B9P> I have a prefect flow that is executi...
# ask-marvin
g
@Marvin I have a prefect flow that is executing in AWS. I have added a healthcheck to the prefect worker, but I would also like to add a healthcheck to the prefect task/flow runner. For the worker it was easy as that was launched with
prefect worker start
and I added the
--with-healthcheck
flag. However, I can't see how to add a healthcheck to the task/flow runner itself. For reference, I had an issue where a failing/blocking async task was causing a complete system hang - the task runner would become completely unresponsive, be unable to be cancelled etc.
m
thought for 224.9 seconds
Short answer: there isn’t a built‑in healthcheck HTTP endpoint for the flow/task runner. The
--with-healthcheck
flag only exists for workers. For individual flow runs, Prefect 3.x relies on heartbeats and state transitions for liveness detection. What you can do instead - Detect and recover from hung runs (recommended) - Enable flow runner heartbeats and use a Prefect Automation to detect “zombie” runs and take action (crash/cancel/notify). - Set an env var on your flow runtime:
PREFECT_RUNNER_HEARTBEAT_FREQUENCY=30
(minimum 30s; default is disabled) - Create the “Detect zombie flows” automation in the UI and choose the action you want (e.g., Change flow run state to Crashed and Send notification). - Docs: Detect zombie flows, settings ref: Runner heartbeat setting - Prevent blocking async tasks from hanging your run - Add timeouts: - Per task:
@task(timeout_seconds=...)
- Per flow:
@flow(timeout_seconds=...)
- For async tasks, wrap long sections with
timeout_async(...)
to ensure cooperative cancellation:
Copy code
from prefect import flow, task
    from prefect.utilities.timeout import timeout_async

    @task(timeout_seconds=300)
    async def do_work():
        async with timeout_async(300):
            # awaited async work here
            ...
- Isolate suspect code in separate processes to hard‑kill if it wedges the interpreter:
Copy code
from prefect import flow, task
    from prefect.task_runners import ProcessPoolTaskRunner

    @task(timeout_seconds=120)
    def cpu_or_blocking_io():
        # blocking library call here
        ...

    @flow(task_runner=ProcessPoolTaskRunner())
    def my_flow():
        fut = cpu_or_blocking_io.submit()
        return fut.result()
Using a ProcessPoolTaskRunner helps when third‑party C extensions or blocking calls ignore/circumvent async cancellation. - If you must expose a container health endpoint - There’s no Prefect‑native endpoint for flow runs. If you’re on Kubernetes/ECS and need a liveness/readiness probe, use one of these patterns: - Sidecar that queries the Prefect API for the flow run’s last heartbeat and returns 200/503 accordingly. This is robust even when your flow process is hung. - A tiny HTTP server started alongside your flow that returns 200 only if a heartbeat file/timestamp is being updated. Note: if the interpreter is fully wedged, that server won’t respond either—sidecar is safer. - Worker healthcheck docs for reference: Worker healthchecks Why cancellation felt unresponsive - In async Python, cancellation is cooperative—code has to hit an await/cancellation point. A blocking or CPU‑bound call in an async task can starve the loop and prevent cancellation from being delivered. Timeouts, process isolation, or moving blocking calls to threads/processes mitigate this. A few clarifying questions so I can tailor concrete examples: - Which AWS runtime are you using for flow runs (ECS, EKS, EC2/Docker, Lambda)? - Are you on Prefect Cloud or OSS? - Would you prefer an example of a Kubernetes/ECS health probe that checks Prefect heartbeats via a sidecar?