Short answer: that message is a generic “the pod is gone” signal. It means the Kubernetes worker checked the Job(s) for your flow run and could not find any pod that is currently Active or has Succeeded. It usually accompanies infrastructure loss (eviction, OOMKill, node/pod deletion, TTL cleanup), and Prefect marks the flow as Crashed. The task runs showing Running are typically stale states that didn’t get a final update before the pod disappeared. What typically causes it - Pod evicted by the scheduler (memory/disk pressure) or node went NotReady - Container OOMKilled (memory limit exceeded) - Pod deleted manually or by a controller - Job/pod cleaned up by TTL after finishing/failed - Namespace mismatch or RBAC preventing the worker from listing pods (less common) - Worker lost track temporarily after a restart (rare; worker is stateless but re-attaches via labels) How to debug quickly 1) Find the job/pod for the flow run - In the Prefect UI, open the flow run and note the

flow_run_id

. - Then in your cluster:

kubectl get jobs -A -l <http://prefect.io/flow-run-id=<FLOW_RUN_ID|prefect.io/flow-run-id=<FLOW_RUN_ID>>
kubectl get pods -A -l <http://prefect.io/flow-run-id=<FLOW_RUN_ID|prefect.io/flow-run-id=<FLOW_RUN_ID>>

2) Inspect the job/pod status and events

kubectl describe job <JOB_NAME>
kubectl describe pod <POD_NAME>
kubectl get events --sort-by=.metadata.creationTimestamp -A

Look for reasons like Evicted, OOMKilled, NodeNotReady, ImagePullBackOff. 3) Check container termination reason and logs

kubectl get pod <POD_NAME> -o jsonpath='{.status.containerStatuses[*].state.terminated.reason}{"\n"}'
kubectl logs job/<JOB_NAME> --timestamps --all-containers
# If the container restarted:
kubectl logs <POD_NAME> --previous

4) Check worker logs at DEBUG If you run the Kubernetes worker yourself, enable debug logging to see reconciliation details:

PREFECT_LOGGING_LEVEL=DEBUG

Then check the worker pod logs to see when it decided the flow crashed and why. 5) Verify job TTL cleanup If pods vanish too quickly to inspect, your job might have

ttlSecondsAfterFinished

set: - Increase TTL or remove it from the work pool’s base job template while you debug, so pods stick around for inspection. - Note: disabling TTL usually means omitting the field, not setting it to 0. Why the parent flow says Crashed but tasks show Running - In Prefect 3.x, a Kubernetes deployment runs your flow and its tasks in the same pod (unless you’re using a distributed task runner like Dask/Ray). If the pod dies unexpectedly, Prefect can’t update each task’s final state and they can remain “Running” while the flow is marked “Crashed.” This is expected after infra loss. Common remediations - If OOMKilled: increase memory requests/limits in the work pool’s base job template, or reduce memory usage. - If evicted: increase requests, reduce node pressure, or use node pools with more headroom; consider Pod Priority if appropriate. - If TTL cleanups are too fast: increase or remove

ttlSecondsAfterFinished

while debugging. - If using spot/preemptible nodes: expect terminations; add retries to the flow, reduce runtime, or add graceful shutdown handling. - Ensure the worker’s namespace/RBAC allows listing pods/jobs in the namespace you’re using. Helpful docs - States overview: Prefect 3.x States - Kubernetes deployments: Prefect on Kubernetes - Logs API: Prefect REST API: Logs - Kubernetes integration: prefect-kubernetes If you can share: - The flow_run_id and namespace - A

kubectl describe pod <pod>

excerpt (especially Events and containerStatuses) - Whether you’re using a distributed task runner

- Your work pool’s base job template (resources + ttlSecondsAfterFinished) …I can help pinpoint the exact cause and suggest the right fix.