https://prefect.io logo
Title
l

Leon Kozlowski

02/02/2023, 7:16 PM
We run prefect agents on EKS and I'm working through upgrading to
2.7.10
I've managed to get a flow deployed, but it seems that the agent can't submit flow runs due to a permissions error (Posting tracebacks in thread)
k8s API error
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:prefect-orion-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
This is what my role looks like:
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: {{ include "prefect-orion-agent.fullname" . }}-rbac
  namespace: {{ .Release.Namespace }}
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
I wonder why the agent is trying to do anything in the
kube-system
ns?
z

Zanie

02/02/2023, 7:21 PM
We read the kube-system namespace to get a unique identifier for the cluster so when you cancel a run we know which cluster is to look in
Unfortunately that’s the only way Kubernetes provides to uniquely identify a cluster 🤦
w

Walter Cavinaw

02/02/2023, 7:25 PM
I believe the workaround is to add PREFECT_KUBERNETES_CLUSTER_UID to the agent environment
z

Zanie

02/02/2023, 7:28 PM
Yes you can provide that if you do not want to give permissions, just set it to a unique identifier for the cluster
l

Leon Kozlowski

02/02/2023, 7:28 PM
Ok cool - thanks guys
n

Nick Coy

02/02/2023, 7:29 PM
@Leon Kozlowski in 2.7.10 running
prefect kubernetes manifest agent
adds a clusterrole which solved that for me
🙌 1
z

Zanie

02/02/2023, 7:29 PM
Our Helm chart performs this lookup at install-time so it doesn’t need permissions
If anyone wants to contribute docs for this I’d appreciate it a ton!
l

Leon Kozlowski

02/02/2023, 7:31 PM
Gotcha, we use a custom helm chart (I didn't have a clusterrole)
I suspect this wouldn't be a problem for users of the prefect created helm chart since the clusterrole is included
z

Zanie

02/02/2023, 7:35 PM
Yeah we have support in our helm chart and the template you generate via our CLI