We run prefect agents on EKS and I'm working throu...
# prefect-community
We run prefect agents on EKS and I'm working through upgrading to
I've managed to get a flow deployed, but it seems that the agent can't submit flow runs due to a permissions error (Posting tracebacks in thread)
k8s API error
Copy code
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:prefect-orion-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
This is what my role looks like:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
  name: {{ include "prefect-orion-agent.fullname" . }}-rbac
  namespace: {{ .Release.Namespace }}
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
I wonder why the agent is trying to do anything in the
We read the kube-system namespace to get a unique identifier for the cluster so when you cancel a run we know which cluster is to look in
Unfortunately that’s the only way Kubernetes provides to uniquely identify a cluster 🤦
I believe the workaround is to add PREFECT_KUBERNETES_CLUSTER_UID to the agent environment
Yes you can provide that if you do not want to give permissions, just set it to a unique identifier for the cluster
Ok cool - thanks guys
@Leon Kozlowski in 2.7.10 running
prefect kubernetes manifest agent
adds a clusterrole which solved that for me
🙌 1
Our Helm chart performs this lookup at install-time so it doesn’t need permissions
If anyone wants to contribute docs for this I’d appreciate it a ton!
Gotcha, we use a custom helm chart (I didn't have a clusterrole)
I suspect this wouldn't be a problem for users of the prefect created helm chart since the clusterrole is included
Yeah we have support in our helm chart and the template you generate via our CLI