https://prefect.io logo
Join SlackCommunities
Powered by
Hello, I would like to optimize my K8 based work ...
# ask-community
v
Hello, I would like to optimize my K8 based work pool removing KubernetesClusterConfig property. K8 work pool works fine with the property. The documentation tells "Specify a KubernetesClusterConfig block to configure the Kubernetes cluster for job creation. In most cases, leave the cluster config blank since the worker should already have appropriate access and permissions. We recommend using this setting when deploying a worker to a cluster that differs from the one executing the flow runs". When I remove the property, I receive the error message below and my flow gets Crashed status but somehow it can be completed successfully with Completed status. What can cause this behavior and how to fix it?
Copy code
Reported flow run '25c4d89c-09c0-4672-a141-ba3001f339ba' as crashed: Flow run could not be submitted to infrastructure:
ApiException()

Failed to submit flow run '25c4d89c-09c0-4672-a141-ba3001f339ba' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 1011, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 602, in run
    pid = await self._get_infrastructure_pid(job, client)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 820, in _get_infrastructure_pid
    cluster_uid = await self._get_cluster_uid(client)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 857, in _get_cluster_uid
    namespace = await v1.read_namespace("kube-system")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
    raise e
  File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
    response_data = await self.request(
                    ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 210, in GET
    return (await self.request("GET", url,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 204, in request
    raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id': '20b71fa0-1d6e-4bf7-9851-c7657c7df58f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '01512e26-1e08-4454-826c-5123b9b4312b', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ef45a754-ef89-4256-b665-96a9b632a469', 'Date': 'Wed, 05 Mar 2025 21:28:44 GMT', 'Content-Length': '363')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:xxx-dev-prefect:xxx-prefect-worker\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}

Worker 'KubernetesWorker d6d28a0d-b17b-4435-aeb6-44a96fbd3fff' submitting flow run '25c4d89c-09c0-4672-a141-ba3001f339ba'
Thank you.
n
hi @Vitaly Saversky
Copy code
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:xxx-dev-prefect:xxx-prefect-worker\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
this is a common stumbling block where most often the 
namespace
on the work pool is set to the wrong namespace
looks like its trying to use this one?
Copy code
kube-system
v
Hi @Nate Thank you for your prompt response. I checked the work pool and the namespace is xxx-dev-prefect.
n
hmm, and that's the same one as the service account?
v
Yes, we have the same namespace for both service account and work pool.
I can see the pod was created in xxx-dev-prefect namespace.
n
are you using the helm chart to run the worker? this should be handled by the helm chart
v
Yes, we use Prefect Helm chart with our value.yaml file
n
can you try setting 
PREFECT_KUBERNETES_CLUSTER_UID
in the 
env
section of your values.yaml? to any value not quite sure why this would be necessary, but curious if this fixes it
v
Let me try it.
Hi @Nate Your suggestion worked out. I stopped receiving the error message and it works as expected. This what I added
Copy code
extraEnvVars:
    - name: PREFECT_KUBERNETES_CLUSTER_UID
      value: "workaround_with_KubernetesClusterConfig"
What would be next steps? I consider it as a good workaround. What would be a permanent solution? Are you going to have any plans based on my output? Thank you for your help!
n
we will update this soon so this manual action is no longer required - thanks for the update!
v
👍
5 Views
Open in Slack
PreviousNext
Powered by