Hi all Getting an issue that Im hoping to get some help on I Prefect Community #prefect-kubernetes

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"prefect\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

The kubernetes work pool has the namespace set to

prefect

and the service account name set to

prefect-worker

. At first I was able to run jobs and it was working but then I had to make edits to the role to allow it to read secrets and then this started happening. Yaml files I used for the Roles are in the thread! Would really appreciate any insight into this - thanks!

✅ 1

Kiley Roberson

08/02/2023, 7:23 PM

Copy code

kind: Role
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  namespace: prefect
  name: prefect-worker
rules:
- apiGroups: [""]
  resources: ["jobs.batch", "pods/log", "pods/status", "pods"]
  verbs: ["get", "watch", "list", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: prefect-worker
  namespace: prefect
subjects:
- kind: ServiceAccount
  name: prefect-worker
  namespace: prefect
roleRef:
  kind: Role
  name: prefect-worker
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>

Kiley Roberson

08/02/2023, 7:24 PM

Copy code

% kubectl describe role prefect-worker
Name:         prefect-worker
Labels:       <http://app.kubernetes.io/component=worker|app.kubernetes.io/component=worker>
              <http://app.kubernetes.io/instance=prefect-worker|app.kubernetes.io/instance=prefect-worker>
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/name=prefect-worker|app.kubernetes.io/name=prefect-worker>
              <http://helm.sh/chart=prefect-worker-2023.07.07|helm.sh/chart=prefect-worker-2023.07.07>
              prefect-version=2.10.20-python3.11-kubernetes
Annotations:  <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: prefect-worker
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: prefect
PolicyRule:
  Resources    Non-Resource URLs  Resource Names  Verbs
  ---------    -----------------  --------------  -----
  jobs.batch   []                 []              [get watch list create update patch delete]
  pods/log     []                 []              [get watch list create update patch delete]
  pods/status  []                 []              [get watch list create update patch delete]
  pods         []                 []              [get watch list create update patch delete]
  secrets      []                 []              [get]

Nate

08/02/2023, 7:45 PM

hey @Kiley Roberson - ~~how have you created your deployment that you're trying to run when you get this error?~~

Jamie Zieziula

08/02/2023, 7:53 PM

i believe these are the permission grants you need

Copy code

- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/status"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]

Jamie Zieziula

08/02/2023, 7:53 PM

i think it may have something to do with you referencing

jobs.batch

rather than have the

batch

be specified in the apiGroup

Kiley Roberson

08/02/2023, 8:08 PM

Amazing! Thanks Jamie that fixed it 😄

🎉 2

2 Views

Open in Slack

Previous Next