Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

We run prefect agents on EKS and I'm working through upgrading to `2.7.10`
I've managed to get a flow deployed, but it seems that the agent can't submit flow runs due to a permissions error
(Posting tracebacks in thread)

k8s API error
```HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:prefect-orion-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}```


This is what my role looks like:
```apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: {{ include "prefect-orion-agent.fullname" . }}-rbac
  namespace: {{ .Release.Namespace }}
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]```


I wonder why the agent is trying to do anything in the  `kube-system` ns?

We read the kube-system namespace to get a unique identifier for the cluster so when you cancel a run we know which cluster is to look in

Unfortunately that’s the only way Kubernetes provides to uniquely identify a cluster :face_palm:

Details in these pull requests <https://github.com/PrefectHQ/prefect/pulls?q=is%3Apr+kube-system>

I believe the workaround is to add PREFECT_KUBERNETES_CLUSTER_UID to the agent environment

Yes you can provide that if you do not want to give permissions, just set it to a unique identifier for the cluster

<@U024EMRLBCH> in 2.7.10 running `prefect kubernetes manifest agent` adds a clusterrole which solved that for me

Our Helm chart performs this lookup at install-time so it doesn’t need permissions

<https://github.com/PrefectHQ/prefect-helm/blob/db5aa040fdac13c74e2616e5a3cd523345d64954/charts/prefect-agent/templates/_helpers.tpl#L24-L35>

If anyone wants to contribute docs for this I’d appreciate it a ton!

Gotcha, we use a custom helm chart (I didn't have a clusterrole)

I suspect this wouldn't be a problem for users of the prefect created helm chart since the clusterrole is included

Yeah we have support in our helm chart and the template you generate via our CLI