@Marvin With either the role

prefect-worker

as created from the helm chart or using a custom role with full api access I still get this error. Are there any other thoughts on a workaround? I've tried reinstalling all charts in the namespace. EKS Version: 1.32

rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]

File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 247, in POST
    return (await self.request("POST", url,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 204, in request
    raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: <CIMultiDictProxy('Audit-Id': '6b5e3e80-e821-4a01-85d2-5a63c7277fe2', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Wed, 19 Feb 2025 03:39:39 GMT', 'Content-Length': '129')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Kube API Log

{"kind":"Event","apiVersion":"audit.k8s.io/v1","status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}}

Prefect Worker.yaml

worker:
  config:
    baseJobTemplate:
      configuration: |
         ...
              "service_account_name": {
                "type": "string",
                "title": "Service Account Name",
                "default": "prefect-full-access",
                "description": "The Kubernetes service account to use for job creation."
          ...
          "job_configuration": {
            "job_manifest": {
              "kind": "Job",
              "spec": {
                "template": {
                  "spec": {
                    "serviceAccountName": "{{ service_account_name }}"

Your message was too long by 444 tokens...

Your message was too long by 33 tokens...

The issue you're experiencing with the Kubernetes job authorization could be related to a few different aspects based on the gathered information. Here are some steps and considerations to help troubleshoot and potentially resolve the issue: 1. Service Account and Role Binding: Ensure that the service account

prefect-full-access

is correctly bound to a role that has the necessary permissions. Even though you've given it full API access, ensure that the role binding is properly configured and associated with the correct namespace. 2. Kubernetes API Access: Double-check that the Kubernetes API server is accessible from the environment where the Prefect worker is running. Network policies or security groups might be restricting access. 3. Token Authentication: Verify that the token used by the service account is valid and correctly mounted in the Prefect worker pod. You can check the service account's secret and ensure it's properly referenced in the pod. 4. Prefect Configuration: Make sure that the Prefect worker is configured to use the correct service account. This involves verifying the

serviceAccountName

in your Helm chart or Kubernetes manifest. 5. API Key and Prefect Cloud: If you're using Prefect Cloud, ensure that the API key is correctly configured and has the necessary permissions to interact with the cloud workspace. 6. Helm Chart Configuration: Review your Helm chart values to ensure that all configurations are correct, including the namespace, service account, and any other relevant settings. For more detailed guidance, you can refer to the following resources from the Prefect documentation: - Kubernetes Infrastructure Examples - Managing Service Accounts in Prefect If these steps don't resolve the issue, consider checking the Kubernetes API logs for more detailed error messages or consult the Prefect community forums for further assistance.