pradana adnyana
12/17/2025, 7:47 AMMarvin
12/17/2025, 7:48 AMMarvin
12/17/2025, 7:53 AMactiveDeadlineSeconds, Docker/container timeouts, serverless max runtimes).
Best practices for long-running work
- Add explicit timeouts/retries where you want upper bounds
- Persist results so you can resume/retry without redoing everything
- Break work into smaller checkpointed tasks for observability and resilience
- Log progress periodically
- Ensure your underlying infrastructure won’t kill the job early
Minimal example
from prefect import flow, task, get_run_logger
import time
@task(
persist_result=True,
timeout_seconds=3600, # 1 hour per chunk
retries=2,
retry_delay_seconds=300
)
def process_chunk(i: int):
logger = get_run_logger()
for m in range(60):
<http://logger.info|logger.info>(f"Chunk {i}: minute {m}")
time.sleep(60)
return f"done-{i}"
@flow(
persist_result=True,
timeout_seconds=172800 # 48 hours total cap (optional)
)
def long_run():
return [process_chunk(i) for i in range(48)]
if __name__ == "__main__":
long_run()
Key notes
- Prefect marks runs “TimedOut” if they exceed timeout_seconds. Cancellation is cooperative; if code is stuck in non-interruptible work, it may not stop immediately.
- Workers have heartbeats (for worker health), not per-task heartbeats; long tasks are fine as long as the worker process stays healthy.
- Make sure your infra timeouts aren’t shorter than your intended run time (e.g., K8s activeDeadlineSeconds).
Helpful docs
- Timeouts and running flows: Write & run workflows
- Persisting results and storage: Results & storage
- Workers overview: Workers
- Worker healthchecks (for reliability): Worker healthchecks
If you can share a bit more:
- How long do you expect the run to last (hours/days)?
- Where will it run (Kubernetes, Docker, VM, serverless)?
- Can the work be chunked into checkpoints, or must it be a single monolithic task?
- Do you need resumability if it’s interrupted?