https://prefect.io
Join Slack
We run prefect agents on EKS and I'm working through upgrading to `2.7.10` I've managed to get a flo...
# ask-community
l
We run prefect agents on EKS and I'm working through upgrading to 
2.7.10
I've managed to get a flow deployed, but it seems that the agent can't submit flow runs due to a permissions error (Posting tracebacks in thread)
k8s API error
Copy code
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:prefect-orion-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
This is what my role looks like:
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: {{ include "prefect-orion-agent.fullname" . }}-rbac
  namespace: {{ .Release.Namespace }}
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/status"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
I wonder why the agent is trying to do anything in the 
kube-system
ns?
z
We read the kube-system namespace to get a unique identifier for the cluster so when you cancel a run we know which cluster is to look in
Unfortunately that’s the only way Kubernetes provides to uniquely identify a cluster 🤦
w
I believe the workaround is to add PREFECT_KUBERNETES_CLUSTER_UID to the agent environment
z
Yes you can provide that if you do not want to give permissions, just set it to a unique identifier for the cluster
l
Ok cool - thanks guys
n
@Leon Kozlowski in 2.7.10 running 
prefect kubernetes manifest agent
adds a clusterrole which solved that for me
🙌 1
z
Our Helm chart performs this lookup at install-time so it doesn’t need permissions
If anyone wants to contribute docs for this I’d appreciate it a ton!
l
Gotcha, we use a custom helm chart (I didn't have a clusterrole)
I suspect this wouldn't be a problem for users of the prefect created helm chart since the clusterrole is included
z
Yeah we have support in our helm chart and the template you generate via our CLI
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by