Thanks for the report — that error usually means the worker hit a problem streaming logs from the Kubernetes pod running your flow’s image. The flow itself may still run; it’s often a logging/permissions/timeout issue rather than a hard failure. A quick triage path and how to tell if it’s K8s/node vs Prefect config: 1) Are runs succeeding but logs are truncated? - Likely a log streaming issue (RBAC/timeout/connection) rather than node failure. 2) Are pods crashing/evicted on the node? - Likely a node/resource issue (OOMKilled, Evicted, NodeUnreachable). Common root causes and fixes - Missing RBAC to read pod logs - The worker’s service account must be able to get pods and pods/log in the namespace where jobs run. Missing pods/log is the most common cause. - Quick check (replace placeholders):

kubectl auth can-i get pods/log --as=system:serviceaccount:<namespace>:<service-account> -n <job-namespace>
    kubectl auth can-i get pods --as=system:serviceaccount:<namespace>:<service-account> -n <job-namespace>

- If needed, grant:

apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
    kind: ClusterRole
    metadata:
      name: prefect-worker
    rules:
    - apiGroups: [""]
      resources: ["pods","pods/log"]
      verbs: ["get","list","watch"]
    - apiGroups: ["batch"]
      resources: ["jobs"]
      verbs: ["create","get","list","watch","delete","patch"]
    ---
    apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
    kind: ClusterRoleBinding
    metadata:
      name: prefect-worker
    roleRef:
      apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
      kind: ClusterRole
      name: prefect-worker
    subjects:
    - kind: ServiceAccount
      name: <service-account>
      namespace: <namespace>

- API watch/log stream timeouts - The worker streams logs via the K8s API; if the stream goes idle or logs expire, you’ll see this error. - Increase watch timeouts in your Kubernetes work pool’s base job template:

{
  "job_configuration": {
    "stream_output": true,
    "pod_watch_timeout_seconds": 120,
    "job_watch_timeout_seconds": 300
  }
}

- Apply via UI (edit the work pool’s job template) or CLI by saving JSON and running:

prefect work-pool update <your-pool> --base-job-template ./template.json

- Container ended before logs were read (very short runs, restarts, or pod deletion) - Check the pod and container state:

kubectl get pods -n <ns> -l <http://prefect.io/flow-run-id=<flow-run-id|prefect.io/flow-run-id=<flow-run-id>>
    kubectl describe pod <pod> -n <ns>
    kubectl logs <pod> -n <ns> --previous

- If you see OOMKilled/Evicted, address resources; otherwise the logging error is often benign. - Node/resource pressure or networking drops - Look for node pressure and evictions:

kubectl top nodes
    kubectl top pods -A
    kubectl get events -A --sort-by='.lastTimestamp' | tail -50

- Ensure your worker can keep a stable connection to the API server (long-lived connections). Prefect’s K8s worker enables TCP keepalive by default. Turn on more diagnostics - Raise Prefect’s logging to see the underlying K8s client error/code:

# Worker environment
  PREFECT_LOGGING_LEVEL=DEBUG
  PREFECT_LOGGING_LOGGERS_PREFECT_KUBERNETES_LEVEL=DEBUG

- Then review the worker logs around the error for clues like 401/403 (RBAC), 410 (expired watch), or connection timeouts. What I’d like from you to pinpoint it - Prefect version and prefect-kubernetes version - Work pool type (Kubernetes?) and the job namespace/service account used - Whether runs succeed but logs cut off, or the entire run fails - A

kubectl describe pod

and

kubectl logs

(or

--previous

if it restarted) from an affected run - Any 401/403/410 codes in the worker logs after enabling DEBUG as above If helpful, I can review your work pool’s base job template to make sure timeouts and

stream_output

are set correctly. References

- Kubernetes integration docs: https://prefecthq.github.io/prefect-kubernetes/ - Logging configuration (Prefect 3.x): https://docs.prefect.io/latest/guides/logging/ Note: In Prefect 3.x, adjust work pool behavior via the base job template (UI or CLI via

--base-job-template

). There aren’t per-flag CLI options for the Kubernetes timeouts; they live in the job configuration.