https://prefect.io logo
Title
l

Lucien Fregosi

01/04/2023, 8:25 AM
hi 👋 When upgrading to
prefect 2.7.5
in our self hosted on kubernetes prefect I got the following error when starting a flow
Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b79d317b-f68f-46ff-b226-09adb2d37b66', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'b89b22ad-0116-43a3-aa7b-0d04dc752da1', 'X-Kubernetes-Pf-Prioritylevel-Uid': '07475109-1722-457e-8c56-1612dc7046b5', 'Date': 'Wed, 04 Jan 2023 08:21:16 GMT', 'Content-Length': '349'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect-conf-lfo\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
Nothing has changed in the cluster or service account and i I rollback to prefect
2.7.0
it works well Any idea / thoughts ?
m

Mason Menges

01/04/2023, 11:09 PM
Hey @Lucien Fregosi I'm not an expert on kubernetes but I believe this is actually related to some changes we made in a recent version https://github.com/PrefectHQ/prefect-helm/issues/91, relevant pr to address this https://github.com/PrefectHQ/prefect/pull/7864
l

Lucien Fregosi

01/05/2023, 9:42 AM
@Mason Menges thanks for the input
j

Jean-Michel Provencher

01/09/2023, 10:10 PM
Isn't that an important breaking change ?
It broke our Kubernetes Agent and I cannot find any relevant public documentation on what the new RBAC needs to be to work with the latest version. https://discourse.prefect.io/t/how-to-deploy-a-prefect-2-0-agent-to-a-local-kubernetes-cluster-and-connect-it-to-cloud-2-0-backend/979
m

Mason Menges

01/09/2023, 11:22 PM
Hey @Jean-Michel Provencher Thanks for bringing that up I raised it with the team so we can get the documentation updated https://github.com/PrefectHQ/prefect/issues/8120
j

Jean-Michel Provencher

01/10/2023, 1:23 PM
Thank you, I'll rollback in the meantime I guess
n

Nick Coy

01/12/2023, 1:14 AM
I also ran into this isssue
👍 2