https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I just upgraded Prefect from 3.4.20 t...
# ask-marvin
d
@Marvin I just upgraded Prefect from 3.4.20 to 3.6.2 and now my Prefect workers are throwing the following error:
Copy code
kopf._cogs.clients.errors.APIForbiddenError: ('pods is forbidden: User "system:serviceaccount:ns-team-arte-prefect:prefect-service-account" cannot list resource "pods" in API group "" at the cluster scope', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'pods is forbidden: User "system:serviceaccount:ns-team-arte-prefect:prefect-service-account" cannot list resource "pods" in API group "" at the cluster scope', 'reason': 'Forbidden', 'details': {'kind': 'pods'}, 'code': 403})
was there a change with how Prefect interacts with Kubernetes?
m
✅ thought for 450.9 seconds
Thanks for the details. Short answer: yes — recent Prefect 3.x releases added/improved the Kubernetes “observer” that uses Kopf to watch Pods/Jobs. By default, the observer runs cluster-wide. When Kopf establishes a watch, Kubernetes performs a “list” at that scope first. If your service account does not have cluster‑scope “list” on Pods, you’ll see the 403 you’re getting. Why you’re seeing it after upgrading - In 3.5.x/3.6.x we shipped enhancements to the Kubernetes observer and pod‑event replication. If the observer is enabled and not namespace‑scoped, Kopf will watch cluster‑wide and will require: - core: pods — list, watch (list is needed to start the watch) - batch: jobs — list, watch - If your RBAC is namespace‑scoped (Role/RoleBinding) and you don’t give cluster‑wide permissions, the observer’s cluster‑wide list will be forbidden. You have three straightforward options 1) Scope the observer to your namespace(s) so you don’t need cluster‑wide RBAC Set this on the worker container (or wherever your worker runs):
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES=ns-team-arte-prefect
This makes Kopf watch only that namespace. With that, a namespaced Role with list/watch permissions is sufficient. 2) Keep cluster-wide observation and grant cluster‑scope RBAC If you want to watch all namespaces, give the worker’s service account cluster‑wide list/watch:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: prefect-worker
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get","create","patch","delete","list","watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
  name: prefect-worker
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: ClusterRole
  name: prefect-worker
subjects:
- kind: ServiceAccount
  name: prefect-service-account
  namespace: ns-team-arte-prefect
3) Disable the observer entirely (if you don’t need pod/job event watching or event replication)
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false
# or, to keep the observer but skip sending Prefect events:
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_REPLICATE_POD_EVENTS=false
Notes - Kubernetes watch semantics require list + watch. Even if you only “watch,” the API starts with a list to get a resourceVersion. - If you scope the observer with 
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES
, ensure your namespaced Role includes list/watch on pods and jobs, e.g.:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-worker
  namespace: ns-team-arte-prefect
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get","create","patch","delete","list","watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-worker
  namespace: ns-team-arte-prefect
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-worker
subjects:
- kind: ServiceAccount
  name: prefect-service-account
  namespace: ns-team-arte-prefect
Docs and references - Kubernetes deployment guide: Prefect on Kubernetes - Prefect Kubernetes integration: prefect-kubernetes docs - API reference: prefect_kubernetes reference - Helm chart (RBAC examples): Prefect Helm repo
If you share how your worker is deployed (namespace-only vs. cluster-wide) and your current Role/ClusterRole, I can recommend the minimal change to resolve the 403 quickly.
d
my worker is deployed namespace-only
@Marvin my worker is deployed namespace-only
m
✅ thought for 36.0 seconds
Got it — since your worker is namespace-only, the fix is to scope the observer to that namespace and ensure your Role has list/watch at the namespace scope. What to do 1) Scope the observer to your namespace Add this env var to your worker Pod/Deployment and restart:
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES=ns-team-arte-prefect
Without this, the observer defaults to cluster-wide and Kubernetes will perform a cluster-scope “list” when establishing watches, which your SA is blocking. 2) Ensure your namespace Role includes list/watch If you haven’t already, add list/watch on pods and jobs for the worker’s service account in your namespace:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: prefect-worker
  namespace: ns-team-arte-prefect
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get","create","patch","delete","list","watch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-worker
  namespace: ns-team-arte-prefect
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-worker
subjects:
- kind: ServiceAccount
  name: prefect-service-account
  namespace: ns-team-arte-prefect
3) Optional: disable the observer if you don’t need it
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false
(or keep it on but disable event replication)
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_REPLICATE_POD_EVENTS=false
Why this is happening - The Kubernetes observer (enabled by default) uses Kopf to watch Pods/Jobs. Establishing a watch starts with a list. If not namespace-scoped, that list is cluster-wide and will be forbidden by a namespace-only service account. Docs and references - Kubernetes deployment guide: Prefect on Kubernetes - Prefect Kubernetes integration: prefect-kubernetes docs - API reference: prefect_kubernetes reference - Helm chart (RBAC examples): Prefect Helm repo If you can share your current Role/RoleBinding and how you’re injecting env vars into the worker Deployment, I can tailor a minimal patch for your setup.
d
@Marvin here is the role and rolebinding:
Copy code
apiVersion: v1
items:
- apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: Role
  metadata:
    labels:
      <http://app.kubernetes.io/component|app.kubernetes.io/component>: role
      <http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-role
      <http://app.kubernetes.io/version|app.kubernetes.io/version>: 3.6.2
      prefect-version: 3.6.2
    name: prefect-role
  rules:
  - apiGroups:
    - ""
    resources:
    - events
    - pods
    - pods/log
    - pods/status
    verbs:
    - get
    - watch
    - list
  - apiGroups:
    - batch
    resources:
    - jobs
    verbs:
    - get
    - list
    - watch
    - create
    - update
    - patch
    - delete
kind: List
metadata:
  resourceVersion: ""
---
apiVersion: v1
items:
- apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: RoleBinding
  metadata:
    labels:
      <http://app.kubernetes.io/component|app.kubernetes.io/component>: rolebinding
      <http://app.kubernetes.io/name|app.kubernetes.io/name>: prefect-role-binding
      <http://app.kubernetes.io/version|app.kubernetes.io/version>: 3.6.2
      prefect-version: 3.6.2
    name: prefect-role-binding
  roleRef:
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
    kind: Role
    name: prefect-role
  subjects:
  - kind: ServiceAccount
    name: prefect-service-account
kind: List
metadata:
  resourceVersion: ""
m
✅ thought for 23.2 seconds
Thanks for sharing the RBAC — that Role looks correct for a namespace-only worker: it has get/list/watch on pods and jobs within the namespace. Your 403 is happening because the Kubernetes observer is running cluster-wide by default, so Kopf starts a cluster-scope “list” when establishing the watch. With a namespaced Role, that’s forbidden. What to change - Scope the observer to your namespace and restart the worker:
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES=ns-team-arte-prefect
- With that set, your existing Role is sufficient because it already includes: - pods: get, list, watch - jobs: get, list, watch, create, update, patch, delete - pods/log: get - Double-check your RoleBinding subject includes the namespace of the ServiceAccount to avoid any ambiguity:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-role-binding
  namespace: ns-team-arte-prefect
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-role
subjects:
- kind: ServiceAccount
  name: prefect-service-account
  namespace: ns-team-arte-prefect
Optional: If you don’t need the observer at all, disable it:
Copy code
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false
(Note: disabling just event replication won’t stop the watch; to avoid watch/list entirely, disable the observer.) Docs - Prefect on Kubernetes - prefect-kubernetes docs - prefect_kubernetes reference If you tell me how you’re deploying the worker (Helm vs. a Deployment manifest), I can share an exact patch snippet to add the env var.
d
@Marvin I'm using a deployment manifest
m
✅ thought for 35.9 seconds
Great — here’s the quickest way to scope the observer to your namespace using your Deployment manifest. Option A: one-liner to add the env var
Copy code
kubectl -n ns-team-arte-prefect set env deploy/<your-deployment-name> PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES=ns-team-arte-prefect
kubectl -n ns-team-arte-prefect rollout status deploy/<your-deployment-name>
Option B: edit your Deployment YAML Add this under the worker container’s 
env:
and re-apply:
Copy code
spec:
  template:
    spec:
      containers:
        - name: <your-worker-container-name>
          env:
            - name: PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES
              value: ns-team-arte-prefect
Then:
Copy code
kubectl apply -f <your-deployment.yaml>
kubectl -n ns-team-arte-prefect rollout status deploy/<your-deployment-name>
Optional: disable the observer entirely
Copy code
kubectl -n ns-team-arte-prefect set env deploy/<your-deployment-name> PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false
Verify - Check logs after rollout: the 403 “cannot list pods at the cluster scope” should disappear. - The worker should still run jobs and stream logs normally with your existing Role. RBAC sanity check Make sure your RoleBinding is namespaced and binds to the SA in the same namespace (add explicit namespace fields if missing):
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
  name: prefect-role-binding
  namespace: ns-team-arte-prefect
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: Role
  name: prefect-role
subjects:
- kind: ServiceAccount
  name: prefect-service-account
  namespace: ns-team-arte-prefect
References - Prefect on Kubernetes - prefect-kubernetes docs - prefect_kubernetes reference If you share your Deployment name and container name, I can give you an exact 
kubectl
command tailored to your manifest.
d
@Marvin does 
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES
need to be set on the prefect server or the prefect jobs config?
m
✅ thought for 29.3 seconds
Set it on the worker pod/container. The Kubernetes observer runs inside the Prefect Kubernetes worker process, so it reads 
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES
from the worker’s environment at startup. - Not needed on Prefect Server/Cloud. - Not needed in your flow-run job pods (those don’t run the observer). Quick patch to your Deployment manifest:
Copy code
spec:
  template:
    spec:
      containers:
        - name: <your-worker-container-name>
          env:
            - name: PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES
              value: ns-team-arte-prefect
Then re-apply and roll out:
Copy code
kubectl apply -f <your-deployment.yaml>
kubectl -n ns-team-arte-prefect rollout status deploy/<your-deployment-name>
Optional alternative: disable the observer entirely on the worker with 
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false
if you don’t need the watch.
d
@Marvin I set 
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_NAMESPACES=ns-team-arte-prefect
and 
PREFECT_INTEGRATIONS_KUBERNETES_OBSERVER_ENABLED=false
, I don't see any errors in the logs but none of my prefect jobs will run. They all just hang with this message:
Copy code
kubectl -n ns-team-arte-prefect logs job.batch/prefect-job-tw5q8
20:59:02.033 | INFO    | prefect.flow_runs.runner - Opening process...
m
✅ thought for 244.7 seconds
d
@Nate when you have a few minutes, I wouldn't mind getting your input 🙂
m
Thanks for the update. The observer settings you changed won’t prevent jobs from running. The “Opening process...” line means the job container started Prefect and is about to spawn the flow subprocess; if nothing follows, the child process isn’t starting or its output isn’t getting through. Let’s quickly pinpoint what’s blocking it. Quick checks 1) Confirm what’s running inside the job pod - Are there one or two Python processes?
Copy code
kubectl -n ns-team-arte-prefect exec job/<job-name> -- ps aux
If you only see one Python process, the subprocess didn’t start. 2) Confirm Prefect version inside the job image
Copy code
kubectl -n ns-team-arte-prefect exec job/<job-name> -- python -c "import prefect; print(prefect.__version__)"
Please share the output. Ideally this matches your worker’s Prefect version (3.6.2). 3) Check the job’s image and command
Copy code
kubectl -n ns-team-arte-prefect get job <job-name> -o yaml
- Which image is used? - What’s the container command/args? (Prefect should be starting the engine subprocess.) 4) Verify the job has API connectivity - Does the pod have 
PREFECT_API_URL
(and 
PREFECT_API_KEY
if using Cloud)?
Copy code
kubectl -n ns-team-arte-prefect exec job/<job-name> -- printenv | grep PREFECT_API
- Basic DNS/egress check (replace host as needed):
Copy code
kubectl -n ns-team-arte-prefect exec job/<job-name> -- sh -c 'apk add --no-cache curl >/dev/null 2>&1 || true; curl -sS -m 5 https://<your-prefect-api-host>/'
If you’re using Prefect Server, try hitting its /api/health endpoint; if Cloud, just confirm DNS resolves. 5) Resources Insufficient CPU/memory can stall process creation. Check container events and usage:
Copy code
kubectl -n ns-team-arte-prefect describe pod <pod-name>
Look for OOMKilled, throttling, or probe failures. Ensure requests/limits give the pod enough headroom (e.g., requests: cpu 250m/mem 512Mi; limits: cpu 1/mem 2Gi). Turn on more logging for the job subprocess Add this env var to the job so we can see more: - If you control deployments, add to the deployment’s job variables/env:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG
- Or, if you want to test quickly, add it to your work pool’s base job template so it propagates to new jobs. Two quick hypotheses to validate - Version/image mismatch: Worker upgraded to 3.6.2 but the job image is still on an older Prefect or missing dependencies, and the subprocess never starts. - Network/DNS: The child process starts but immediately blocks trying to reach the Prefect API or your code source (e.g., Git). With DEBUG logs or “ps aux”, we’ll see whether the child PID exists. Next actions - Please share: - The output of steps 1–4 - The image name used by the job - Whether your flow code is baked into the image or pulled from source (e.g., Git) at runtime - If you’d like, I can also propose a minimal tweak to your work pool/deployment to inject DEBUG logging and a small startup echo to confirm the child starts. References - Kubernetes guide: Prefect on Kubernetes - Kubernetes integration: prefect-kubernetes docs - API reference: prefect_kubernetes reference
n
i dont have time to look at this right now, but whats important is your prefect-kubernetes version diff
we added kopf observer at some point, if that's requesting something your infra doesn't expect in a breaking way, please open an issue
👍 1
49 Views
Open in Slack
PreviousNext
Powered by