<@ULVA73B9P> What best practice to prevent a long ...
# ask-marvin
b
@Marvin What best practice to prevent a long running flow if something gets stuck? Is there a timeout I can set at the deployment level or should I set up an automation to cancel/rerun
m
thought for 118.6 seconds
Great question. In Prefect 3.x, the best practice is to combine code-level timeouts with an automation that handles truly “stuck” (zombie) runs. Short answer - There is no deployment-level timeout knob. Set timeouts at the flow/task level. - Add an Automation to detect missing heartbeats and cancel/rerun if a run becomes a zombie. What to do 1) Set timeouts in code (graceful, retry-friendly) - Flow-level timeout applies to every run of that flow (including runs from a deployment):
Copy code
from prefect import flow, task

@flow(timeout_seconds=3600, retries=1, retry_delay_seconds=60)
def my_flow():
    ...
- Task-level timeout caps long I/O or external calls and can retry independently:
Copy code
@task(timeout_seconds=300, retries=2, retry_delay_seconds=[30, 60])
def fetch_data(...):
    ...
- For fine-grained control around a specific block of code:
Copy code
from prefect.utilities.timeout import timeout, timeout_async

with timeout(seconds=120):
    blocking_call()

# or for async:
# async with timeout_async(seconds=120):
#     await blocking_call()
Notes: - On timeout, the run enters a TimedOut failure state. If you’ve set
retries
, Prefect will retry automatically. - Links: Timeout utilities 2) Add an Automation to catch “zombie” runs (no heartbeat) - Prefect runners emit heartbeats (default every 30s). If a run is “Running” but stops heartbeating, it may be stuck/crashed. - Create an Automation with a proactive trigger that expects a heartbeat or terminal state within a window (e.g., 90–180s). Actions: - Cancel the flow run - Optionally trigger a rerun (Run deployment) - Links: - Detect zombie flows - Event triggers and Automations Optional infrastructure-level guards - If you run on Kubernetes or other schedulers, you can set job-level limits (e.g., K8s
activeDeadlineSeconds
) to hard-kill very long runs. Prefer Prefect timeouts for graceful failure/retries; infra-level kills will look like crashes to Prefect. When to use which - Use flow/task timeouts for known upper bounds and to enable clean retries. - Use the zombie-detection Automation to catch deadlocks, crashes, or environment failures where the process stops heartbeating. If you share your worker type (Docker, K8s, Process) and how long your flows typically run, I can suggest concrete timeout values and a step-by-step Automation setup.