Vadym Dytyniak
12/09/2022, 10:34 AM10:28:28.260 | ERROR | prefect.agent - Failed to submit flow run 'a7e88141-ff19-4c87-90d5-48a82995d75f' to infrastructure. │
│ Traceback (most recent call last): │
│ File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 417, in _submit_run_and_capture_errors │
│ result = await infrastructure.run(task_status=task_status) │
│ File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 277, in run │
│ pid = await run_sync_in_worker_thread(self._get_infrastructure_pid, job) │
│ File "/usr/local/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 69, in run_sync_in_worker_thread │
│ return await anyio.to_thread.run_sync(call, cancellable=True) │
│ File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync │
│ return await get_asynclib().run_sync_in_worker_thread( │
│ File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread │
│ return await future │
│ File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run │
│ result = context.run(func, *args) │
│ File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 359, in _get_infrastructure_pid │
│ cluster_uid = self._get_cluster_uid() │
│ File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/kubernetes.py", line 384, in _get_cluster_uid │
│ namespace = client.read_namespace("kube-system") │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 22476, in read_namespace │
│ return self.read_namespace_with_http_info(name, **kwargs) # noqa: E501 │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 22555, in read_namespace_with_http_info │
│ return self.api_client.call_api( │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api │
│ return self.__call_api(resource_path, method, │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api │
│ response_data = self.request( │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 373, in request │
│ return self.rest_client.GET(url, │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 241, in GET
│ return self.request("GET", url, │
│ File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 235, in request │
│ raise ApiException(http_resp=r) │
│ kubernetes.client.exceptions.ApiException: (403) │
│ Reason: Forbidden │
│ HTTP response headers: HTTPHeaderDict({'Audit-Id': 'fd00e1d8-48df-4c8d-8c08-835bc03928f7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options │
│ ': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '58043781-9ba3-433c-8746-28fbf1be655d', 'X-Kubernetes-Pf-Prioritylevel-Uid': '5862da58-0cf2-4368-ba81-8ac98e0a2c38', 'Date': 'Fri, 09 Dec 20 │
│ 22 10:28:28 GMT', 'Content-Length': '357'}) │
│ HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect-agents:pref │
│ ect-agent-dev\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
Andreas Nigg
12/09/2022, 11:22 AMVadym Dytyniak
12/09/2022, 11:25 AMAndreas Nigg
12/09/2022, 11:38 AM---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
name: prefect-ns-watcher
namespace: prefect
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
name: prefect-ns-watcher-role-binding
namespace: prefect
subjects:
- kind: ServiceAccount
name: prefect-orion-agent
roleRef:
kind: ClusterRole
name: prefect-ns-watcher
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
Disclaimer: Please make sure to check the security implications on your side, before applying the fix.
(I allow myself to also post this answer to the channel, as it might help the one or the other)Zanie
12/09/2022, 3:42 PMAndreas Nigg
12/09/2022, 8:40 PMZanie
12/09/2022, 8:42 PMPREFECT_KUBERNETES_CLUSTER_UID=…
Andreas Nigg
12/09/2022, 8:49 PMZanie
12/09/2022, 8:52 PMJames Zhang
12/17/2022, 12:49 PMAndreas Nigg
12/17/2022, 12:56 PMJames Zhang
12/17/2022, 12:57 PMAndreas Nigg
12/17/2022, 1:09 PMJames Zhang
12/17/2022, 1:12 PMprefect kubernetes manifest agent
right?metadata.uid
in the kube-system
namespace manifest
2. yesPeyton Runyan
12/17/2022, 3:07 PMAleksandr Liadov
01/13/2023, 3:44 PM