https://prefect.io logo
Join Slack
Channels
Powered by
Hello - I've deployed server, worker on azure k8s,...
# ask-community
p
Hello - I've deployed server, worker on azure k8s, deployed a flow on to it. While Im trying to run the flow, Im getting following error. Any pointers please. Trace is added to the thread.
Copy code
Failed to submit flow run 'a130952c-4667-49e3-9dc8-f055ea6e579d' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/workers/base.py", line 908, in _submit_run_and_capture_errors
    result = await self.run(
             ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 581, in run
    pid = await run_sync_in_worker_thread(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 136, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 859, in _get_infrastructure_pid
    cluster_uid = self._get_cluster_uid(client)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect_kubernetes/worker.py", line 907, in _get_cluster_uid
    namespace = core_client.read_namespace("kube-system")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 22686, in read_namespace
    return self.read_namespace_with_http_info(name, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 22765, in read_namespace_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a2f49be7-5c23-4cbd-8f6e-6ca0b17df112', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e1308394-e92b-41df-bedf-22a010c83151', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'f1660392-31a8-47c7-a82e-ff6f173d865f', 'Date': 'Thu, 20 Jun 2024 18:58:09 GMT', 'Content-Length': '391'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:prefect:prefect-worker\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\": Azure does not have opinion for this user.","reason":"Forbidden","details":{"name":"kube-system","kind":"namespaces"},"code":403}
j
hi! if you set the value for clusteruuid, this should solve your issue!
p
Hi Jamie - I resolved this by creating service account and binding it with cluster admin role
j
Hi Parash, that’s fine - but we wouldn’t recommend you give the worker pod cluster admin! You can also resolve the error by setting the key i mentioned above
p
Sure Jamie. Thank you so much!
Have we documented these permissions required for the worker pod? I did not find any docs on service account
j
Hi Parash - I don't know that we explicitly document why access to the kube-system is needed but here is some context
Copy code
There is no real unique identifier for a cluster. However, the `kube-system`
        namespace is immutable and has a persistence UID that we use instead.

        PREFECT_KUBERNETES_CLUSTER_UID can be set in cases where the `kube-system`
        namespace cannot be read e.g. when a cluster role cannot be created. If set,
        this variable will be used and we will not attempt to read the `kube-system`
        namespace.
so in the case of the helm chart, we (the chart) try to grab the kube-system namespace UID at install time, but sometimes that doesn't work. Instead of asking users to grant elevated access (cluster admin) to the worker, we provide an alternative method, which is by providing the helm chart value 
worker.clusterUid
p
@Antoine Tehio tagging you here to check the suggestion by Jamie. Thank you Jamie, I will try this out.
🎉 2
a
Thank you @Parash I will try this too.
m
@Jamie Zieziula Sorry for jumping in this thread. I just had a question about the workaround suggested here. Can I use a fake value for the clusterUid in the worker values? As far as I can tell, this is just a sanity check when deploying it to the cluster? Thanks!
j
Hi! My guess is probably but I've only ever used the uuid of the namespace
m
I see. Thanks a lot
4 Views
Open in Slack
PreviousNext
Powered by