Thanks for flagging this. That message is coming from the Kubernetes observer in Prefect 3.x. It will mark a flow run as Crashed with “No active or succeeded pods found for any job” when a Kubernetes Job for that flow run has failed (exceeded its backoffLimit) and there are no other active or succeeded Jobs/pods for the same flow run. A few quick clarifying questions to narrow this down: - Which exact versions are you on now? (Prefect and prefect-kubernetes integration) - Are you running with the Kubernetes worker, and if so, what’s your work pool’s base job template (especially

spec.backoffLimit

)? - Do these crashes coincide with pods being OOMKilled, Evicted, or ImagePullBackOff? - Were any of these flow runs paused/suspended? Fast triage steps - Inspect the flow run to confirm state history:

prefect flow-run inspect <FLOW_RUN_ID>

- Check related Jobs for the flow run and their status:

kubectl get jobs -A -l <http://prefect.io/flow-run-id=<FLOW_RUN_ID|prefect.io/flow-run-id=<FLOW_RUN_ID>> -o wide
kubectl describe job <JOB_NAME>

- Check pods for the job and look for OOMKilled/Evicted/ImagePull errors:

kubectl get pods -A -l job-name=<JOB_NAME> -o wide
kubectl describe pod <POD_NAME>
kubectl logs <POD_NAME> --all-containers=true --timestamps

- Look at your worker/observer logs around the time of the crash for context:

kubectl logs -l <http://prefect.io/component=kubernetes-worker|prefect.io/component=kubernetes-worker> --all-containers=true --tail=200

Common causes and mitigations - Job backoff exhausted: if

status.failed > spec.backoffLimit

, K8s marks the job failed and the observer will crash the run if there aren’t any other active/succeeded jobs for that flow run. - Mitigation: raise

spec.backoffLimit

in your work pool’s base job template to allow more retries on transient failures. - Pod-level issues (OOMKilled/Evicted/ImagePullBackOff/CrashLoopBackOff): - Mitigation: adjust container resources (requests/limits), image pull secrets/permissions, node placement, or fix the underlying app error. - Replacement job lag: the observer will briefly wait for another job to appear for the same flow run. If a replacement isn’t created in time (e.g., worker restart), it may crash the run. - Mitigation: ensure your worker reliably resubmits jobs promptly; check worker logs for re-submission timing. - Paused/Suspended flows: there was a recent fix in the Kubernetes observer to avoid incorrectly crashing paused/suspended flow runs. - Mitigation: if you use flow suspension, make sure you’re on the latest Prefect + prefect-kubernetes versions. Temporary workaround while you investigate - You can disable the Kubernetes observer so it won’t auto-mark runs as Crashed:

export PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false

Re-enable it once you’ve resolved the underlying job/pod behavior. Helpful version checks

prefect version
python -c "import prefect; print('prefect', prefect.__version__); import prefect_kubernetes; print('prefect-kubernetes', prefect_kubernetes.__version__)"

If you can share: - Prefect and prefect-kubernetes versions - A recent flow run id - Your work pool base job template (especially

spec.backoffLimit

and container resources) - A sample Job/pod describe output showing the failure reason …I can pinpoint whether this is expected observer behavior, a configuration tweak, or a bug we should escalate.