<@ULVA73B9P> How long can I have parent flows run ...
# ask-marvin
m
@Marvin How long can I have parent flows run for? I'm having issues where if my flow runs for 4 or more hours it will always report as crashed in the dashboard, even though the pod is still running and creating children (subflows). All my work pool definitions are correct, it is not timing out, the job and pod are still active. Perhaps it has something to do with kubelet, but this is something prefect should be handling, what solutions do you purpose?
m
thought for 774.5 seconds
Short answer: there’s no hard limit. In Prefect 2.x a parent flow can run indefinitely unless you set
timeout_seconds
on the flow, your Kubernetes job has an
activeDeadlineSeconds
, or the infrastructure/worker times out its watch. Given your symptoms (parent flow marked Crashed after ~4h while the pod keeps running and still launches subflows), this almost always points to the Kubernetes-side watcher timing out or the worker process restarting, not to a Prefect-imposed runtime cap. What typically causes this in 2.x - Kubernetes worker/job watch timeouts: In 2.x, the Kubernetes runner/worker watches the Job/Pod. If the watch hits its timeout, the worker stops monitoring and returns a failure which Prefect records as Crashed — even though the pod may still be running. - Variables to check on your Kubernetes work pool:
job_watch_timeout_seconds
and
pod_watch_timeout_seconds
. If these are low or hit an upper bound (or a bugged default), you’ll see the behavior you described. In some environments ~4h correlates with watch connections expiring multiple times and not being re-established properly. - Worker restarts: If your Kubernetes worker pod restarts (liveness probe, OOM, HPA/KEDA scale-in) while it’s watching a long-running job, the worker can report a crash for the flow while the job/pod continues. - Network/ingress idle timeouts: Long-lived connections (e.g., watches, log streams) can be cut by an ingress/load balancer. Some older plugin versions treat these as fatal instead of reconnecting gracefully. What I recommend for Prefect 2.x 1) Raise or disable the Kubernetes job watch timeout - In your Kubernetes work pool variables, increase
job_watch_timeout_seconds
substantially (e.g., 86400 for 24h) or set it to “no timeout” if the UI allows null/blank. Also consider raising
pod_watch_timeout_seconds
if pod startup is slow. - If you’re using the older
KubernetesJob
infrastructure block (not the worker), its
job_watch_timeout_seconds
default is None (no timeout); if you set it, a timeout will mark the run failed/crashed while the job keeps running. 2) Upgrade your Kubernetes integration - Make sure you’re on the latest 2.x-compatible
prefect-kubernetes
package. Older versions had brittle watch handling; newer releases reconnect more reliably to the Kubernetes API after a watch timeout. 3) Verify your worker stability - Check if the worker pod restarts around the 4h mark: -
kubectl get pods -n <ns>
and
kubectl describe pod <worker-pod>
for restarts/events - Loosen liveness/readiness probes or increase resources if they’re causing restarts - If you run workers behind an ingress/proxy, increase idle timeouts to exceed your longest flow. 4) Avoid relying on log/watch activity to “prove liveness” - If the parent flow mostly waits for child runs to finish (quiet periods), the K8s watch may go quiet too. Either: - Increase
job_watch_timeout_seconds
so quiet periods don’t trip the timeout, or - Add a tiny keepalive in the parent (e.g., a periodic no-op/log task) to ensure regular activity while it waits. 5) Sanity checks on cluster defaults - Confirm no
activeDeadlineSeconds
is being injected into your job/pod by a policy or template. - Ensure there’s no cluster policy killing or evicting long-running control pods (workers or your flow pods). A few details that will help me pinpoint the exact cause - Exact Prefect versions: core/server or Cloud, and
prefect-kubernetes
version - Work pool type and variables (especially
job_watch_timeout_seconds
,
pod_watch_timeout_seconds
, and any TTL/cleanup settings) - The Crashed state message shown for the parent run in the UI (that text is very diagnostic) - Worker logs around the crash time (look for “Job did not complete within timeout” or similar) - Whether the worker pod shows restarts near that time Note on Prefect 3.x
- 3.x adds flow-run heartbeats and server-side zombie detection, which makes these situations much easier to detect and handle. Since you plan to upgrade next month, we can set you up for a smooth transition after we stabilize 2.x on your new infra. If you share the details above, I can give you an exact configuration change (and sample YAML for your work pool) to stop the premature crashes.
m
thought for 120.8 seconds
Short answer: yes — that PR directly targets the behavior you described. What it fixes - The websocket used to “watch” a flow run (e.g.,
prefect deployment run --watch
) would eventually give up on long-running runs after hitting a fixed number of reconnection attempts. This PR resets the reconnection counter after a successful reconnect, so transient/discontinuous network drops over long durations no longer cause the watch to time out. Where it applies - Affects the event subscriber used by: - CLI watching (e.g.,
--watch
on flow runs) - Live event/log streaming paths (also relevant to UI watch behavior) Version availability - The fix was merged for Prefect 3.x and is not backported to 2.x. Since you’re on 2.x right now, you won’t see the improvement until you upgrade. Links - PR: Fix websocket timeout when watching long-running flow runs - Related issue (context): Intermittent websocket disconnects while watching runs If you need a 2.x stopgap until your 3.x upgrade: - Avoid
--watch
for very long runs and check status in the UI or by polling. - If you control the load balancer/proxy in your new infra, increasing idle timeouts or ensuring regular websocket keepalives can reduce disconnects. If you can share your 2.x version and how you’re watching (CLI vs UI) plus any LB/proxy in front of the API, I can suggest more targeted workarounds.