<@ULVA73B9P> can prefect flow use for realy long r...
# ask-marvin
p
@Marvin can prefect flow use for realy long running task?
m
thought for 327.7 seconds
Short answer: yes. In Prefect 3.x you can run flows and tasks for hours or even days. There’s no hard runtime limit imposed by Prefect itself. The main things to watch are timeouts you set and any limits from your infrastructure (e.g., Kubernetes pod
activeDeadlineSeconds
, Docker/container timeouts, serverless max runtimes). Best practices for long-running work - Add explicit timeouts/retries where you want upper bounds - Persist results so you can resume/retry without redoing everything - Break work into smaller checkpointed tasks for observability and resilience - Log progress periodically - Ensure your underlying infrastructure won’t kill the job early Minimal example
Copy code
from prefect import flow, task, get_run_logger
import time

@task(
    persist_result=True,
    timeout_seconds=3600,        # 1 hour per chunk
    retries=2,
    retry_delay_seconds=300
)
def process_chunk(i: int):
    logger = get_run_logger()
    for m in range(60):
        <http://logger.info|logger.info>(f"Chunk {i}: minute {m}")
        time.sleep(60)
    return f"done-{i}"

@flow(
    persist_result=True,
    timeout_seconds=172800       # 48 hours total cap (optional)
)
def long_run():
    return [process_chunk(i) for i in range(48)]

if __name__ == "__main__":
    long_run()
Key notes - Prefect marks runs “TimedOut” if they exceed
timeout_seconds
. Cancellation is cooperative; if code is stuck in non-interruptible work, it may not stop immediately. - Workers have heartbeats (for worker health), not per-task heartbeats; long tasks are fine as long as the worker process stays healthy. - Make sure your infra timeouts aren’t shorter than your intended run time (e.g., K8s
activeDeadlineSeconds
). Helpful docs - Timeouts and running flows: Write &amp; run workflows - Persisting results and storage: Results &amp; storage - Workers overview: Workers - Worker healthchecks (for reliability): Worker healthchecks If you can share a bit more: - How long do you expect the run to last (hours/days)? - Where will it run (Kubernetes, Docker, VM, serverless)? - Can the work be chunked into checkpoints, or must it be a single monolithic task? - Do you need resumability if it’s interrupted?