hi 👋 When upgrading to

prefect 2.7.5

in our self hosted on kubernetes prefect I got the following error when starting a flow

Submission failed. kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b79d317b-f68f-46ff-b226-09adb2d37b66', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'b89b22ad-0116-43a3-aa7b-0d04dc752da1', 'X-Kubernetes-Pf-Prioritylevel-Uid': '07475109-1722-457e-8c56-1612dc7046b5', 'Date': 'Wed, 04 Jan 2023 08:21:16 GMT', 'Content-Length': '349'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect-conf-lfo\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}

Nothing has changed in the cluster or service account and i I rollback to prefect

2.7.0

it works well Any idea / thoughts ?

Hey @Lucien Fregosi I'm not an expert on kubernetes but I believe this is actually related to some changes we made in a recent version https://github.com/PrefectHQ/prefect-helm/issues/91, relevant pr to address this https://github.com/PrefectHQ/prefect/pull/7864

@Mason Menges thanks for the input

Isn't that an important breaking change ?

Hey @Jean-Michel Provencher Thanks for bringing that up I raised it with the team so we can get the documentation updated https://github.com/PrefectHQ/prefect/issues/8120

Thank you, I'll rollback in the meantime I guess

I also ran into this isssue