https://prefect.io logo
Channels
Powered by
k

Kiley Roberson

08/02/2023, 7:23 PM
Hi all! Getting an issue that Im hoping to get some help on. Im getting this error when running a flow:
Copy code
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}
The kubernetes work pool has the namespace set to 
prefect
and the service account name set to 
prefect-worker
. At first I was able to run jobs and it was working but then I had to make edits to the role to allow it to read secrets and then this started happening. Yaml files I used for the Roles are in the thread! Would really appreciate any insight into this - thanks!
✅ 1
Copy code
kind: Role
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  namespace: prefect
  name: prefect-worker
rules:
- apiGroups: [""]
  resources: ["jobs.batch", "pods/log", "pods/status", "pods"]
  verbs: ["get", "watch", "list", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: prefect-worker
  namespace: prefect
subjects:
- kind: ServiceAccount
  name: prefect-worker
  namespace: prefect
roleRef:
  kind: Role
  name: prefect-worker
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Copy code
% kubectl describe role prefect-worker
Name:         prefect-worker
Labels:       <http://app.kubernetes.io/component=worker|app.kubernetes.io/component=worker>
              <http://app.kubernetes.io/instance=prefect-worker|app.kubernetes.io/instance=prefect-worker>
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/name=prefect-worker|app.kubernetes.io/name=prefect-worker>
              <http://helm.sh/chart=prefect-worker-2023.07.07|helm.sh/chart=prefect-worker-2023.07.07>
              prefect-version=2.10.20-python3.11-kubernetes
Annotations:  <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: prefect-worker
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: prefect
PolicyRule:
  Resources    Non-Resource URLs  Resource Names  Verbs
  ---------    -----------------  --------------  -----
  jobs.batch   []                 []              [get watch list create update patch delete]
  pods/log     []                 []              [get watch list create update patch delete]
  pods/status  []                 []              [get watch list create update patch delete]
  pods         []                 []              [get watch list create update patch delete]
  secrets      []                 []              [get]
n

Nate

08/02/2023, 7:45 PM
hey @Kiley Roberson - how have you created your deployment that you're trying to run when you get this error?
j

Jamie Zieziula

08/02/2023, 7:53 PM
i believe these are the permission grants you need
Copy code
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
i think it may have something to do with you referencing 
jobs.batch
rather than have the 
batch
be specified in the apiGroup
k

Kiley Roberson

08/02/2023, 8:08 PM
Amazing! Thanks Jamie that fixed it 😄
🎉 2
Powered by