Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
pacc-london-2023
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
l
Leon Kozlowski
02/02/2023, 7:16 PMWe run prefect agents on EKS and I'm working through upgrading to
2.7.10k8s API error
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:default:prefect-orion-agent\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}This is what my role looks like:
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
name: {{ include "prefect-orion-agent.fullname" . }}-rbac
namespace: {{ .Release.Namespace }}
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/status"]
verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]I wonder why the agent is trying to do anything in the
kube-systemz
Zanie
02/02/2023, 7:21 PMWe read the kube-system namespace to get a unique identifier for the cluster so when you cancel a run we know which cluster is to look in
Unfortunately that’s the only way Kubernetes provides to uniquely identify a cluster 🤦
Details in these pull requests https://github.com/PrefectHQ/prefect/pulls?q=is%3Apr+kube-system
w
Walter Cavinaw
02/02/2023, 7:25 PMI believe the workaround is to add PREFECT_KUBERNETES_CLUSTER_UID to the agent environment
z
Zanie
02/02/2023, 7:28 PMYes you can provide that if you do not want to give permissions, just set it to a unique identifier for the cluster
l
Leon Kozlowski
02/02/2023, 7:28 PMOk cool - thanks guys
n
Nick Coy
02/02/2023, 7:29 PM@Leon Kozlowski in 2.7.10 running
prefect kubernetes manifest agent🙌 1
z
Zanie
02/02/2023, 7:29 PMOur Helm chart performs this lookup at install-time so it doesn’t need permissions
If anyone wants to contribute docs for this I’d appreciate it a ton!
l
Leon Kozlowski
02/02/2023, 7:31 PMGotcha, we use a custom helm chart (I didn't have a clusterrole)
I suspect this wouldn't be a problem for users of the prefect created helm chart since the clusterrole is included
z
Zanie
02/02/2023, 7:35 PMYeah we have support in our helm chart and the template you generate via our CLI