https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am using self hosted prefect deploy...
# ask-marvin
d
@Marvin I am using self hosted prefect deployment on GKE in GCP with a worker-pool of type kubernetes. I am using basic auth. When I enable basic auth, my k8s jobs generated by the Runs give a forbidden error accessing the prefect api. When I disable the basic auth, the jobs succeed . What do I need to configure in the worker-pool, or in the worker, or in the deployment, or in the flows to make it work ?
m
✅ thought for 96.8 seconds
Here's the complete solution to fix the authentication issue: 1. Configure Basic Auth in the Worker Pool Base Job Template Create a base job template for your Kubernetes worker pool that includes the basic auth credentials. You'll need to add these as environment variables:
Copy code
yaml
   # base-job-template.yaml
   job_configuration:
     env:
       PREFECT_API_URL: "<http://your-prefect-server:4200/api>"
       PREFECT_API_USERNAME: "${PREFECT_API_USERNAME}"
       PREFECT_API_PASSWORD: "${PREFECT_API_PASSWORD}"
2. Create/Update the Work Pool
Copy code
bash
   prefect work-pool create "your-pool-name" \
     --type kubernetes \
     --base-job-template base-job-template.yaml
3. Configure the Worker When starting your worker, ensure it has the basic auth credentials:
Copy code
bash
   prefect worker start \
     --pool "your-pool-name" \
     --username your-username \
     --password your-password
4. Use Kubernetes Secrets (Recommended) Instead of passing credentials directly, it's better to use Kubernetes secrets:
Copy code
yaml
   # Create a Kubernetes secret
   apiVersion: v1
   kind: Secret
   metadata:
     name: prefect-auth
   type: Opaque
   data:
     username: <base64-encoded-username>
     password: <base64-encoded-password>
Then modify your base job template to use these secrets:
Copy code
yaml
   job_configuration:
     env:
       - name: PREFECT_API_USERNAME
         valueFrom:
           secretKeyRef:
             name: prefect-auth
             key: username
       - name: PREFECT_API_PASSWORD
         valueFrom:
           secretKeyRef:
             name: prefect-auth
             key: password
5. Verify Configuration - Make sure the 
PREFECT_API_URL
points to your Prefect server - Ensure the credentials match those configured in your Prefect server - Check that the Kubernetes service account used by the worker has access to read the secrets 6. Debug Tips - You can check the logs of the failed Kubernetes jobs to see the exact authentication error - Verify the environment variables are correctly set in the pods by running:
Copy code
bash
     kubectl exec -it <pod-name> -- env | grep PREFECT
- Ensure your Kubernetes network policies (if any) allow the pods to reach the Prefect API If you're still experiencing issues after implementing these changes, could you please share: 1. The exact error message from the Kubernetes job logs 2. The current base job template configuration (with sensitive information redacted) 3. Whether you're using any custom network policies in your GKE cluster This will help me provide more specific guidance for your setup.
d
@Marvin The error is this: prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'http://my-prefect-url/api/flow_runs/2b035aee-2f8e-45a8-8591-e31ab03b0bf7' base job template is this: { "variables": { "type": "object", "properties": { "env": { "type": "object", "title": "Environment Variables", "default": { "PREFECT_API_URL": "http://prefect-url/api", "PREFECT_API_PASSWORD": "*****", "PREFECT_API_USERNAME": "*********" }, "description": "Environment variables to set when starting a flow run.", "additionalProperties": { "anyOf": [ { "type": "string" }, { "type": "null" } ] } }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Name", "description": "Name given to infrastructure created by a worker." }, "image": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Image", "examples": [ "docker.io/prefecthq/prefect:3-latest" ], "description": "The image reference of a container image to use for created jobs. If not set, the latest Prefect image will be used." }, "labels": { "type": "object", "title": "Labels", "description": "Labels applied to infrastructure created by a worker.", "additionalProperties": { "type": "string" } }, "command": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Command", "description": "The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker." }, "namespace": { "type": "string", "title": "Namespace", "default": "prefect", "description": "The Kubernetes namespace to create jobs within." }, "stream_output": { "type": "boolean", "title": "Stream Output", "default": true, "description": "If set, output will be streamed from the job to local standard output." }, "cluster_config": { "anyOf": [ { "$ref": "#/definitions/KubernetesClusterConfig" }, { "type": "null" } ], "description": "The Kubernetes cluster config to use for job creation." }, "finished_job_ttl": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "title": "Finished Job TTL", "description": "The number of seconds to retain jobs after completion. If set, finished jobs will be cleaned up by Kubernetes after the given delay. If not set, jobs will be retained indefinitely." }, "image_pull_policy": { "enum": [ "IfNotPresent", "Always", "Never" ], "type": "string", "title": "Image Pull Policy", "default": "Always", "description": "The Kubernetes image pull policy to use for job containers." }, "service_account_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Service Account Name", "default": "prefect-worker-base", "description": "The Kubernetes service account to use for job creation." }, "job_watch_timeout_seconds": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "title": "Job Watch Timeout Seconds", "description": "Number of seconds to wait for each event emitted by a job before timing out. If not set, the worker will wait for each event indefinitely." }, "pod_watch_timeout_seconds": { "type": "integer", "title": "Pod Watch Timeout Seconds", "default": 60, "description": "Number of seconds to watch for pod creation before timing out." } }, "definitions": { "KubernetesClusterConfig": { "type": "object", "title": "KubernetesClusterConfig", "required": [ "config", "context_name" ], "properties": { "config": { "type": "object", "title": "Config", "description": "The entire contents of a kubectl config file." }, "context_name": { "type": "string", "title": "Context Name", "description": "The name of the kubectl context to use." } }, "description": "Stores configuration for interaction with Kubernetes clusters.\n\nSee 
from_file
for creation.", "secret_fields": [], "block_type_slug": "kubernetes-cluster-config", "block_schema_references": {} } }, "description": "Default variables for the Kubernetes worker.\n\nThe schema for this class is used to populate the 
variables
section of the default\nbase job template." }, "job_configuration": { "env": "{{ env }}", "name": "{{ name }}", "labels": "{{ labels }}", "command": "{{ command }}", "namespace": "{{ namespace }}", "job_manifest": { "kind": "Job", "spec": { "template": { "spec": { "containers": [ { "env": "{{ env }}", "args": "{{ command }}", "name": "prefect-job", "image": "{{ image }}", "imagePullPolicy": "{{ image_pull_policy }}" } ], "completions": 1, "parallelism": 1, "restartPolicy": "Never", "serviceAccountName": "{{ service_account_name }}" } }, "backoffLimit": 0, "ttlSecondsAfterFinished": "{{ finished_job_ttl }}" }, "metadata": { "labels": "{{ labels }}", "namespace": "{{ namespace }}", "generateName": "{{ name }}-" }, "apiVersion": "batch/v1" }, "stream_output": "{{ stream_output }}", "cluster_config": "{{ cluster_config }}", "job_watch_timeout_seconds": "{{ job_watch_timeout_seconds }}", "pod_watch_timeout_seconds": "{{ pod_watch_timeout_seconds }}" } }
m
Your message was too long, here's your message at the allowed limit: The error is this prefect exceptions PrefectHTTPStatusError Client error '401 Unauthorized' for url '<http //my-prefect-url/api/flow_runs/2b035aee-2f8e-45a8-8591-e31ab03b0bf7>' base job template is this { variables { type object properties { env { type object title Environment Variables default { PREFECT_API_URL <http //prefect-url/api> PREFECT_API_PASSWORD pass PREFECT_API_USERNAME ***** } description Environment variables to set when starting a flow run additionalProperties { anyOf [ { type string } { type null } ] } } name { anyOf [ { type string } { type null } ] title Name description Name given to infrastructure created by a worker } image { anyOf [ { type string } { type null } ] title Image examples [ <http //docker io/prefecthq/prefect 3-latest|docker io/prefecthq/prefect 3-latest> ] description The image reference of a container image to use for created jobs If not set the latest Prefect image will be used } labels { type object title Labels description Labels applied to infrastructure created by a worker additionalProperties { type string } } command { anyOf [ { type string } { type null } ] title Command description The command to use when starting a flow run In most cases this should be left blank and the command will be automatically generated by the worker } namespace { type string title Namespace default prefect description The Kubernetes namespace to create jobs within } stream_output { type boolean title Stream Output default true description If set output will be streamed from the job to local standard output } cluster_config { anyOf [ { $ref #/definitions/KubernetesClusterConfig } { type null } ] description The Kubernetes cluster config to use for job creation } finished_job_ttl { anyOf [ { type integer } { type null } ] title Finished Job TTL description The number of seconds to retain jobs after completion If set finished jobs will be cleaned up by Kubernetes after the given delay If not set jobs will be retained indefinitely } image_pull_policy { enum [ IfNotPresent Always Never ] type string title Image Pull Policy default Always description The Kubernetes image pull policy to use for job containers } service_account_name { anyOf [ { type string } { type null } ] title Service Account Name default prefect-worker-base description The Kubernetes service account to use for job creation } job_watch_timeout_seconds { anyOf [ { type integer } { type null } ] title Job Watch Timeout Seconds description Number of seconds to wait for each event emitted by a job before timing out If not set the worker will wait for each event indefinitely } pod_watch_timeout_seconds { type integer title Pod Watch Timeout Seconds default 60 description Number of seconds to watch for pod creation before timing out } } definitions { KubernetesClusterConfig { type object title KubernetesClusterConfig required [ config context_name ] properties { config { type object title Config description The entire contents of a kubectl config file } context_name { type string title Context Name description The name of the kubectl context to use } }
Your message was too long, here's your message at the allowed limit: The error is this prefect exceptions PrefectHTTPStatusError Client error '401 Unauthorized' for url '<http //my-prefect-url/api/flow_runs/2b035aee-2f8e-45a8-8591-e31ab03b0bf7>' base job template is this { variables { type object properties { env { type object title Environment Variables default { PREFECT_API_URL <http //prefect-url/api> PREFECT_API_PASSWORD * PREFECT_API_USERNAME ***** } description Environment variables to set when starting a flow run additionalProperties { anyOf [ { type string } { type null } ] } } name { anyOf [ { type string } { type null } ] title Name description Name given to infrastructure created by a worker } image { anyOf [ { type string } { type null } ] title Image examples [ <http //docker io/prefecthq/prefect 3-latest|docker io/prefecthq/prefect 3-latest> ] description The image reference of a container image to use for created jobs If not set the latest Prefect image will be used } labels { type object title Labels description Labels applied to infrastructure created by a worker additionalProperties { type string } } command { anyOf [ { type string } { type null } ] title Command description The command to use when starting a flow run In most cases this should be left blank and the command will be automatically generated by the worker } namespace { type string title Namespace default prefect description The Kubernetes namespace to create jobs within } stream_output { type boolean title Stream Output default true description If set output will be streamed from the job to local standard output } cluster_config { anyOf [ { $ref #/definitions/KubernetesClusterConfig } { type null } ] description The Kubernetes cluster config to use for job creation } finished_job_ttl { anyOf [ { type integer } { type null } ] title Finished Job TTL description The number of seconds to retain jobs after completion If set finished jobs will be cleaned up by Kubernetes after the given delay If not set jobs will be retained indefinitely } image_pull_policy { enum [ IfNotPresent Always Never ] type string title Image Pull Policy default Always description The Kubernetes image pull policy to use for job containers } service_account_name { anyOf [ { type string } { type null } ] title Service Account Name default prefect-worker-base description The Kubernetes service account to use for job creation } job_watch_timeout_seconds { anyOf [ { type integer } { type null } ] title Job Watch Timeout Seconds description Number of seconds to wait for each event emitted by a job before timing out If not set the worker will wait for each event indefinitely } pod_watch_timeout_seconds { type integer title Pod Watch Timeout Seconds default 60 description Number of seconds to watch for pod creation before timing out } } definitions { KubernetesClusterConfig { type object title KubernetesClusterConfig required [ config context_name ] properties { config { type object title Config description The entire contents of a kubectl config file } context_name { type string title Context Name description The name of the kubectl context to use } }
d
@Marvin The error is this: prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'http://my-prefect-url/api/flow_runs/2b035aee-2f8e-45a8-8591-e31ab03b0bf7' Wait, i'll give you the base job template in the next 2 messages
m
✅ thought for 6.4 seconds
Ah, that 401 Unauthorized error is specifically indicating that the basic authentication credentials are not being properly passed or recognized in the flow run. Let's wait for your base job template to better diagnose the issue, but I can already tell you that this is definitely an authentication problem where the Kubernetes job (flow run) is trying to communicate back to the Prefect API without proper credentials. A 401 error (as opposed to a 403 Forbidden) specifically means the request was made without any authentication credentials or with invalid credentials, rather than having insufficient permissions. Once you share your base job template, I'll be able to: 1. Check how the authentication credentials are being passed 2. Verify the environment variable configuration 3. Identify any potential issues with how the credentials are being propagated to the flow runs Please feel free to redact any sensitive information (URLs, credentials, etc.) when you share the template.
d
@Marvin base jog template part 1 { "variables": { "type": "object", "properties": { "env": { "type": "object", "title": "Environment Variables", "default": { "PREFECT_API_URL": "http://prefect-url/api", "PREFECT_API_PASSWORD": "**********", "PREFECT_API_USERNAME": "*********" }, "description": "Environment variables to set when starting a flow run.", "additionalProperties": { "anyOf": [ { "type": "string" }, { "type": "null" } ] } }, "name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Name", "description": "Name given to infrastructure created by a worker." }, "image": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Image", "examples": [ "docker.io/prefecthq/prefect:3-latest" ], "description": "The image reference of a container image to use for created jobs. If not set, the latest Prefect image will be used." }, "labels": { "type": "object", "title": "Labels", "description": "Labels applied to infrastructure created by a worker.", "additionalProperties": { "type": "string" } }, "command": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Command", "description": "The command to use when starting a flow run. In most cases, this should be left blank and the command will be automatically generated by the worker." }, "namespace": { "type": "string", "title": "Namespace", "default": "prefect", "description": "The Kubernetes namespace to create jobs within." }, "stream_output": { "type": "boolean", "title": "Stream Output", "default": true, "description": "If set, output will be streamed from the job to local standard output." }, "cluster_config": { "anyOf": [ { "$ref": "#/definitions/KubernetesClusterConfig" }, { "type": "null" } ], "description": "The Kubernetes cluster config to use for job creation." }, "finished_job_ttl": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "title": "Finished Job TTL", "description": "The number of seconds to retain jobs after completion. If set, finished jobs will be cleaned up by Kubernetes after the given delay. If not set, jobs will be retained indefinitely." }, "image_pull_policy": { "enum": [ "IfNotPresent", "Always", "Never" ], "type": "string", "title": "Image Pull Policy", "default": "Always", "description": "The Kubernetes image pull policy to use for job containers." }, "service_account_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Service Account Name", "default": "prefect-worker-base", "description": "The Kubernetes service account to use for job creation." }, "job_watch_timeout_seconds": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "title": "Job Watch Timeout Seconds", "description": "Number of seconds to wait for each event emitted by a job before timing out. If not set, the worker will wait for each event indefinitely." }, "pod_watch_timeout_seconds": { "type": "integer", "title": "Pod Watch Timeout Seconds", "default": 60, "description": "Number of seconds to watch for pod creation before timing out." } }, "definitions": { "KubernetesClusterConfig": { "type": "object", "title": "KubernetesClusterConfig", "required": [ "config", "context_name" ], "properties": { "config": { "type": "object", "title": "Config", "description": "The entire contents of a kubectl config file." }, "context_name": { "type": "string", "title": "Context Name",
m
✅ thought for 7.2 seconds
I see the first part of your base job template, and I can already spot a potential issue. The environment variables are being set at the top level, but they need to be properly propagated to the job specification. Let me wait for the second part of the template to see the complete picture, but I can already tell you that the way the authentication credentials are configured might need adjustment. A few things I notice: 1. The environment variables are defined in the 
variables.env
section, but we need to make sure these are actually being passed to the Kubernetes job pods 2. The 
PREFECT_API_URL
format seems to have an extra 
<
and 
>
- it should be a plain URL without these brackets 3. The credentials are hardcoded in the template - it would be better to use references to Kubernetes secrets Please share the second part of the template, and I'll provide a complete solution for properly configuring the authentication.
d
@Marvin base job template part 2 "description": "The name of the kubectl context to use." } }, "description": "Stores configuration for interaction with Kubernetes clusters.\n\nSee 
from_file
for creation.", "secret_fields": [], "block_type_slug": "kubernetes-cluster-config", "block_schema_references": {} } }, "description": "Default variables for the Kubernetes worker.\n\nThe schema for this class is used to populate the 
variables
section of the default\nbase job template." }, "job_configuration": { "env": "{{ env }}", "name": "{{ name }}", "labels": "{{ labels }}", "command": "{{ command }}", "namespace": "{{ namespace }}", "job_manifest": { "kind": "Job", "spec": { "template": { "spec": { "containers": [ { "env": "{{ env }}", "args": "{{ command }}", "name": "prefect-job", "image": "{{ image }}", "imagePullPolicy": "{{ image_pull_policy }}" } ], "completions": 1, "parallelism": 1, "restartPolicy": "Never", "serviceAccountName": "{{ service_account_name }}" } }, "backoffLimit": 0, "ttlSecondsAfterFinished": "{{ finished_job_ttl }}" }, "metadata": { "labels": "{{ labels }}", "namespace": "{{ namespace }}", "generateName": "{{ name }}-" }, "apiVersion": "batch/v1" }, "stream_output": "{{ stream_output }}", "cluster_config": "{{ cluster_config }}", "job_watch_timeout_seconds": "{{ job_watch_timeout_seconds }}", "pod_watch_timeout_seconds": "{{ pod_watch_timeout_seconds }}" } }
m
✅ thought for 19.3 seconds
✅ thought for 24.1 seconds
Ah, now I can see the complete picture and I've identified the issue. The problem is with how the environment variables are being passed to the Kubernetes job. There are a few issues in the template that need to be fixed: 1. The environment variables are being passed incorrectly in the container spec. The current format 
"env": "{{ env }}"
will not work properly because Kubernetes expects a specific format for environment variables. Here's the corrected base job template (I'll focus on the relevant parts that need to change):
Copy code
json
{
  "variables": {
    "type": "object",
    "properties": {
      "env": {
        "type": "object",
        "title": "Environment Variables",
        "default": {
          "PREFECT_API_URL": "<http://prefect-url/api>",
          "PREFECT_API_PASSWORD": "pass",
          "PREFECT_API_USERNAME": "*****"
        },
        "description": "Environment variables to set when starting a flow run.",
        "additionalProperties": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ]
        }
      },
      // ... other variables remain the same ...
    }
  },
  "job_configuration": {
    // ... other configuration remains the same ...
    "job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}",
                "args": "{{ command }}",
                "env": [
                  {
                    "name": "PREFECT_API_URL",
                    "value": "{{ env.PREFECT_API_URL }}"
                  },
                  {
                    "name": "PREFECT_API_USERNAME",
                    "value": "{{ env.PREFECT_API_USERNAME }}"
                  },
                  {
                    "name": "PREFECT_API_PASSWORD",
                    "value": "{{ env.PREFECT_API_PASSWORD }}"
                  }
                ]
              }
            ],
            "completions": 1,
            "parallelism": 1,
            "restartPolicy": "Never",
            "serviceAccountName": "{{ service_account_name }}"
          }
        },
        // ... rest remains the same ...
      }
    }
  }
}
However, a more secure approach would be to use Kubernetes secrets. Here's how to implement that: 1. First, create a Kubernetes secret:
Copy code
bash
kubectl create secret generic prefect-auth \
  --namespace prefect \
  --from-literal=PREFECT_API_USERNAME='your-username' \
  --from-literal=PREFECT_API_PASSWORD='your-password'
2. Then modify the template to use the secrets: ```json { // ... other configuration remains the same ... "job_configuration": { "job_manifest": { "kind": "Job", "spec": { "template": { "spec": { "containers": [ { "name": "prefect-job", "image": "{{ image }}", "imagePullPolicy": "{{ image_pull_policy }}", "args": "{{ command }}", "env": [ { "name": "PREFECT_API_URL", "value": "{{ env.PREFECT_API_URL }}" }, { "name": "PREFECT_API_USERNAME", "valueFrom": { "secretKeyRef": { "name": "prefect-auth", "key": "PREFECT_API_USERNAME" } } }, { "name": "PREFECT_API_PASSWORD", "valueFrom": { "secretKeyRef": { "name": "prefect-auth", "key": "PREFECT_API_PASSWORD" } } } ]
} ] } } } } } }
Copy code
To implement these changes:

1. Save the modified template to a file (e.g., `updated-job-template.json`)

2. Update your work pool with the new template:
```bash
prefect work-pool create "your-pool-name" \
  --type kubernetes \
  --base-job-template updated-job-template.json \
  --overwrite
3. Make sure the service account specified in 
service_account_name
has access to read the secrets in your namespace. The key changes made are: 1. Properly formatted the environment variables section in the Kubernetes job spec 2. Removed the angle brackets (
<>
) from the API URL 3. Added proper structure for environment variables in the container spec 4. (In the secure version) Added proper secret references This should resolve the 401 Unauthorized errors you're seeing, as the authentication credentials will now be properly passed to the flow run pods. Would you like me to explain any part of these changes in more detail?
I notice you've shared the first part again. I'm still missing the job specification part of the template that would show how these variables are actually being used in the Kubernetes job/pod configuration. However, I can already identify several issues and provide a solution based on what I see and the 401 error you're getting. Here are the problems and solutions: 1. URL Format Issue Your 
PREFECT_API_URL
contains extra brackets:
Copy code
json
   "PREFECT_API_URL": "<http://prefect-url/api>"
Should be:
Copy code
json
   "PREFECT_API_URL": "<http://prefect-url/api>"
2. Credentials Security Instead of hardcoding credentials in the template, let's use Kubernetes secrets. First, create a secret:
Copy code
yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: prefect-api-credentials
     namespace: prefect
   type: Opaque
   data:
     username: <base64-encoded-username>
     password: <base64-encoded-password>
3. Updated Base Job Template Here's how your template should look (this includes the job specification part that would properly inject the environment variables):
Copy code
json
   {
     "variables": {
       "type": "object",
       "properties": {
         "env": {
           "type": "object",
           "title": "Environment Variables",
           "default": {}
         },
         // ... other properties as in your original template ...
       }
     },
     "job_configuration": {
       "apiVersion": "batch/v1",
       "kind": "Job",
       "spec": {
         "template": {
           "spec": {
             "containers": [
               {
                 "name": "prefect-job",
                 "env": [
                   {
                     "name": "PREFECT_API_URL",
                     "value": "<http://prefect-url/api>"
                   },
                   {
                     "name": "PREFECT_API_USERNAME",
                     "valueFrom": {
                       "secretKeyRef": {
                         "name": "prefect-api-credentials",
                         "key": "username"
                       }
                     }
                   },
                   {
                     "name": "PREFECT_API_PASSWORD",
                     "valueFrom": {
                       "secretKeyRef": {
                         "name": "prefect-api-credentials",
                         "key": "password"
                       }
                     }
                   }
                 ]
               }
             ]
           }
         }
       }
     }
   }
4. Implementation Steps a. First, create the Kubernetes secret:
Copy code
bash
   kubectl create secret generic prefect-api-credentials \
     --from-literal=username='your-username' \
     --from-literal=password='your-password' \
     -n prefect
b. Update your work pool with the new template:
Copy code
bash
   prefect work-pool create --type kubernetes \
     --base-job-template updated-template.json \
     your-pool-name
Or update existing work pool:
Copy code
bash
   prefect work-pool update --base-job-template updated-template.json your-pool-name
5. Verification Steps After implementing these changes:
Copy code
bash
   # Check if the secret is properly created
   kubectl get secret prefect-api-credentials -n prefect
   
   # When a job is running, check its environment variables
   kubectl exec -it <pod-name> -n prefect -- env | grep PREFECT
Could you please: 1. Confirm if you can share the job specification part of your template (the part that defines how the Kubernetes job/pod is created)? 2. Let me know if you're using any custom Kubernetes network policies? 3. Verify that the service account being used (
prefect-worker-base
in your template) has permissions to read the secrets? This will help me provide more specific guidance if needed.
d
@Marvin the error still persists isn't the env var PREFECT_API_AUTH_STRING ? what am I doing wrong?
m
✅ thought for 42.1 seconds
Yes, you're correct! Here's the fix. You need to modify your base job template to use 
PREFECT_API_AUTH_STRING
instead of separate username/password variables. 1. First, create a Kubernetes secret with the auth string:
Copy code
bash
kubectl create secret generic prefect-api-auth \
  --namespace prefect \
  --from-literal=auth_string='username:password'
2. Then modify your base job template. Here's the critical part that needs to change:
Copy code
json
{
  "variables": {
    "type": "object",
    "properties": {
      "env": {
        "type": "object",
        "title": "Environment Variables",
        "default": {
          "PREFECT_API_URL": "<http://prefect-url/api>"
        }
      },
      // ... rest of your variables ...
    }
  },
  "job_configuration": {
    "job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "{{ image_pull_policy }}",
                "args": "{{ command }}",
                "env": [
                  {
                    "name": "PREFECT_API_URL",
                    "value": "{{ env.PREFECT_API_URL }}"
                  },
                  {
                    "name": "PREFECT_API_AUTH_STRING",
                    "valueFrom": {
                      "secretKeyRef": {
                        "name": "prefect-api-auth",
                        "key": "auth_string"
                      }
                    }
                  }
                ]
              }
            ],
            "serviceAccountName": "{{ service_account_name }}"
          }
        }
      }
    }
  }
}
3. Update your work pool with the new template:
Copy code
bash
prefect work-pool update --base-job-template updated-template.json your-pool-name
Key points: - Remove 
PREFECT_API_USERNAME
and 
PREFECT_API_PASSWORD
- Add 
PREFECT_API_AUTH_STRING
in the format "username:password" - Make sure your 
PREFECT_API_URL
doesn't have the 
<>
brackets To verify the configuration: 1. Check if a pod has the correct environment variables:
Copy code
bash
kubectl exec -it <pod-name> -n prefect -- env | grep PREFECT
2. You should see:
Copy code
PREFECT_API_URL=<http://your-prefect-url/api>
PREFECT_API_AUTH_STRING=username:password
This should resolve the 401 Unauthorized error. Let me know if you need any clarification or if you're still experiencing issues!
d
@Marvin it does not resolve. Still have the error I can confirm the k8s job has properly configuresd the env vars PREFECT_API_URL and PREFECT_API_AUTH_STRING
m
✅ thought for 19.8 seconds
This is very helpful! I see a similar issue has been reported. Let's verify a few critical things: 1. Can you confirm the exact format of your auth string? It should be exactly:
Copy code
username:password
(no spaces, no additional characters) 2. Let's check the version of Prefect you're using:
Copy code
bash
prefect version
3. Can you try to get the exact error message from the Kubernetes job logs? You can use:
Copy code
bash
kubectl logs <pod-name> -n prefect
4. Also, let's verify the auth configuration on your Prefect server. The server should have: - 
PREFECT_SERVER_API_AUTH_STRING
set (not 
PREFECT_API_AUTH_STRING
) - The matching credentials as what you're using in the worker Here's what I suggest trying: 1. First, verify your server configuration:
Copy code
bash
# On the server
echo $PREFECT_SERVER_API_AUTH_STRING
2. Then, try setting both environment variables in your job template:
Copy code
json
"env": [
  {
    "name": "PREFECT_API_URL",
    "value": "{{ env.PREFECT_API_URL }}"
  },
  {
    "name": "PREFECT_API_AUTH_STRING",
    "valueFrom": {
      "secretKeyRef": {
        "name": "prefect-api-auth",
        "key": "auth_string"
      }
    }
  },
  {
    "name": "PREFECT_API_KEY",
    "valueFrom": {
      "secretKeyRef": {
        "name": "prefect-api-auth",
        "key": "auth_string"
      }
    }
  }
]
3. Also, try setting these environment variables directly on a test pod to verify the authentication works:
Copy code
bash
kubectl run prefect-test --image=prefecthq/prefect:2-latest -it --rm \
  --env="PREFECT_API_URL=your-api-url" \
  --env="PREFECT_API_AUTH_STRING=username:password" \
  -- python3 -c "from prefect.client import get_client; client = get_client(); print(client.get('/hello'))"
Could you please: 1. Confirm your Prefect version 2. Share the complete error message from the job logs 3. Verify the server's auth string configuration 4. Try the test with both environment variables set as shown above This will help us narrow down exactly where the authentication is failing.
d
@Marvin prefect version 3.4.10 job logs: 155455.744 | DEBUG | prefect.profiles - Using profile 'ephemeral' 155456.969 | DEBUG | prefect.runner - Starting runner... 155457.011 | DEBUG | prefect.events.clients - Connecting to ws://prefect.qa.gcp.radicant.com/api/events/out 155457.012 | DEBUG | prefect.events.clients - Reconnecting... 155457.052 | DEBUG | prefect.events.clients - pinging... 155457.054 | DEBUG | prefect.events.clients - authenticating... 155457.056 | DEBUG | prefect.events.clients - auth result {'type': 'auth_success'} 155457.057 | DEBUG | prefect.events.clients - filtering events since 2025-08-07 155357.056985+00:00... 155457.059 | DEBUG | prefect.client - Connecting to API at http://prefect.qa.gcp.radicant.com/api/ 155457.059 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 155457.059 | DEBUG | prefect.events.clients - Opening websocket connection. 155457.085 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 155457.086 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 155457.087 | DEBUG | prefect.events.clients - Resending 0 unconfirmed events. 155457.087 | DEBUG | prefect.events.clients - Finished resending unconfirmed events. 155457.088 | DEBUG | prefect.runner - Limit slot acquired for flow run '4f983d8f-795b-4d96-86b9-21d7d0e651de' 155457.108 | DEBUG | prefect.runner - Stopping runner... 155457.111 | DEBUG | prefect.events.clients - Connection closed with "OK" status + Exception Group Traceback (most recent call last): | File "/usr/local/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper | return fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/cli/_types.py", line 156, in sync_fn | return asyncio.run(async_fn(*args, **kwargs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run | return runner.run(main) | ^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run | return self._loop.run_until_complete(task) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete | return future.result() | ^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/cli/flow_run.py", line 412, in execute | await runner.execute_flow_run(id) | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 571, in execute_flow_run | async with context: | ^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 1536, in aexit | await self._exit_stack.__aexit__(*exc_info) | File "/usr/local/lib/python3.12/contextlib.py", line 754, in aexit | raise exc_details[1] | File "/usr/local/lib/python3.12/contextlib.py", line 737, in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 576, in execute_flow_run | flow_run = await self._client.read_flow_run(flow_run_id) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in read_flow_run | response = await self.request( | ^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request | return await self._client.send(request) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 379, in send | response.raise_for_status() | File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 163, in raise_for_status | raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause | prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'http://my-prefect-url.com/api/flow_runs/4f983d8f-795b-4d96-86b9-21d7d0e651de' | Response: {'exception_message': 'Unauthorized'} | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401 +------------------------------------ An exception occurred.
m
Your message was too long, here's your message at the allowed limit: prefect version 3 4 10 job logs 15 54 55 744 | DEBUG | prefect profiles - Using profile 'ephemeral' 15 54 56 969 | DEBUG | prefect runner - Starting runner 15 54 57 011 | DEBUG | prefect events clients - Connecting to <ws //prefect qa gcp radicant com/api/events/out> 15 54 57 012 | DEBUG | prefect events clients - Reconnecting 15 54 57 052 | DEBUG | prefect events clients - pinging 15 54 57 054 | DEBUG | prefect events clients - authenticating 15 54 57 056 | DEBUG | prefect events clients - auth result {'type' 'auth_success'} 15 54 57 057 | DEBUG | prefect events clients - filtering events since 2025-08-07 15 53 57 056985+00 00 15 54 57 059 | DEBUG | prefect client - Connecting to API at <http //prefect qa gcp radicant com/api/> 15 54 57 059 | DEBUG | prefect events clients - Reconnecting websocket connection 15 54 57 059 | DEBUG | prefect events clients - Opening websocket connection 15 54 57 085 | DEBUG | prefect events clients - Pinging to ensure websocket connected 15 54 57 086 | DEBUG | prefect events clients - Pong received Websocket connected 15 54 57 087 | DEBUG | prefect events clients - Resending 0 unconfirmed events 15 54 57 087 | DEBUG | prefect events clients - Finished resending unconfirmed events 15 54 57 088 | DEBUG | prefect runner - Limit slot acquired for flow run '4f983d8f-795b-4d96-86b9-21d7d0e651de' 15 54 57 108 | DEBUG | prefect runner - Stopping runner 15 54 57 111 | DEBUG | prefect events clients - Connection closed with OK status + Exception Group Traceback (most recent call last) | File /usr/local/lib/python3 12/site-packages/prefect/cli/_utilities py line 44 in wrapper | return fn(*args *kwargs) | ^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/cli/_types py line 156 in sync_fn | return asyncio run(async_fn(*args *kwargs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/asyncio/runners py line 195 in run | return runner run(main) | ^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/asyncio/runners py line 118 in run | return self _loop run_until_complete(task) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/asyncio/base_events py line 691 in run_until_complete | return future result() | ^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/cli/flow_run py line 412 in execute | await runner execute_flow_run(id) | File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 571 in execute_flow_run | async with context | ^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 1536 in aexit | await self _exit_stack __aexit__(*exc_info) | File /usr/local/lib/python3 12/contextlib py line 754 in aexit | raise exc_details[1] | File /usr/local/lib/python3 12/contextlib py line 737 in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/anyio/_backends/_asyncio py line 772 in aexit | raise BaseExceptionGroup( | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last) | File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 576 in execute_flow_run | flow_run = await self _client read_flow_run(flow_run_id) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/client/orchestration/_flow_runs/client py line 668 in read_flow_run | response = await self request( | ^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/client/orchestration/base
d
@Marvin prefect version 3.4.10 logs: (wait for the part 2 of the logs in the next message) 155455.744 | DEBUG | prefect.profiles - Using profile 'ephemeral' 155456.969 | DEBUG | prefect.runner - Starting runner... 155457.011 | DEBUG | prefect.events.clients - Connecting to ws://prefect.qa.gcp.radicant.com/api/events/out 155457.012 | DEBUG | prefect.events.clients - Reconnecting... 155457.052 | DEBUG | prefect.events.clients - pinging... 155457.054 | DEBUG | prefect.events.clients - authenticating... 155457.056 | DEBUG | prefect.events.clients - auth result {'type': 'auth_success'} 155457.057 | DEBUG | prefect.events.clients - filtering events since 2025-08-07 155357.056985+00:00... 155457.059 | DEBUG | prefect.client - Connecting to API at http://prefect.qa.gcp.radicant.com/api/ 155457.059 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 155457.059 | DEBUG | prefect.events.clients - Opening websocket connection. 155457.085 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 155457.086 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 155457.087 | DEBUG | prefect.events.clients - Resending 0 unconfirmed events. 155457.087 | DEBUG | prefect.events.clients - Finished resending unconfirmed events. 155457.088 | DEBUG | prefect.runner - Limit slot acquired for flow run '4f983d8f-795b-4d96-86b9-21d7d0e651de' 155457.108 | DEBUG | prefect.runner - Stopping runner... 155457.111 | DEBUG | prefect.events.clients - Connection closed with "OK" status + Exception Group Traceback (most recent call last): | File "/usr/local/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper | return fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/cli/_types.py", line 156, in sync_fn | return asyncio.run(async_fn(*args, **kwargs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run | return runner.run(main) | ^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run | return self._loop.run_until_complete(task) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete | return future.result() | ^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/cli/flow_run.py", line 412, in execute | await runner.execute_flow_run(id) | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 571, in execute_flow_run | async with context: | ^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 1536, in aexit | await self._exit_stack.__aexit__(*exc_info) | File "/usr/local/lib/python3.12/contextlib.py", line 754, in aexit | raise exc_details[1] | File "/usr/local/lib/python3.12/contextlib.py", line 737, in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 576, in execute_flow_run | flow_run = await self._client.read_flow_run(flow_run_id) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in read_flow_run | response = await self.request( | ^^^^^^^^^^^^^^^^^^^
m
Your message was too long, here's your message at the allowed limit: prefect version 3 4 10 logs (wait for the part 2 of the logs in the next message) 15 54 55 744 | DEBUG | prefect profiles - Using profile 'ephemeral' 15 54 56 969 | DEBUG | prefect runner - Starting runner 15 54 57 011 | DEBUG | prefect events clients - Connecting to <ws //prefect qa gcp radicant com/api/events/out> 15 54 57 012 | DEBUG | prefect events clients - Reconnecting 15 54 57 052 | DEBUG | prefect events clients - pinging 15 54 57 054 | DEBUG | prefect events clients - authenticating 15 54 57 056 | DEBUG | prefect events clients - auth result {'type' 'auth_success'} 15 54 57 057 | DEBUG | prefect events clients - filtering events since 2025-08-07 15 53 57 056985+00 00 15 54 57 059 | DEBUG | prefect client - Connecting to API at <http //prefect qa gcp radicant com/api/> 15 54 57 059 | DEBUG | prefect events clients - Reconnecting websocket connection 15 54 57 059 | DEBUG | prefect events clients - Opening websocket connection 15 54 57 085 | DEBUG | prefect events clients - Pinging to ensure websocket connected 15 54 57 086 | DEBUG | prefect events clients - Pong received Websocket connected 15 54 57 087 | DEBUG | prefect events clients - Resending 0 unconfirmed events 15 54 57 087 | DEBUG | prefect events clients - Finished resending unconfirmed events 15 54 57 088 | DEBUG | prefect runner - Limit slot acquired for flow run '4f983d8f-795b-4d96-86b9-21d7d0e651de' 15 54 57 108 | DEBUG | prefect runner - Stopping runner 15 54 57 111 | DEBUG | prefect events clients - Connection closed with OK status + Exception Group Traceback (most recent call last) | File /usr/local/lib/python3 12/site-packages/prefect/cli/_utilities py line 44 in wrapper | return fn(*args *kwargs) | ^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/cli/_types py line 156 in sync_fn | return asyncio run(async_fn(*args *kwargs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/asyncio/runners py line 195 in run | return runner run(main) | ^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/asyncio/runners py line 118 in run | return self _loop run_until_complete(task) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/asyncio/base_events py line 691 in run_until_complete | return future result() | ^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/cli/flow_run py line 412 in execute | await runner execute_flow_run(id) | File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 571 in execute_flow_run | async with context | ^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 1536 in aexit | await self _exit_stack __aexit__(*exc_info) | File /usr/local/lib/python3 12/contextlib py line 754 in aexit | raise exc_details[1] | File /usr/local/lib/python3 12/contextlib py line 737 in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/anyio/_backends/_asyncio py line 772 in aexit | raise BaseExceptionGroup( | ExceptionGroup unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last) | File /usr/local/lib/python3 12/site-packages/prefect/runner/runner py line 576 in execute_flow_run | flow_run = await self _client read_flow_run(flow_run_id) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File /usr/local/lib/python3 12/site-packages/prefect/client/orchestration/_flow_runs/client py line 668 in read_flow_run |
d
@Marvin prefect version 3.4.10 logs: (wait for the part 2 of the logs in the next message) 155455.744 | DEBUG | prefect.profiles - Using profile 'ephemeral' 155456.969 | DEBUG | prefect.runner - Starting runner... 155457.011 | DEBUG | prefect.events.clients - Connecting to ws://prefect.qa.gcp.radicant.com/api/events/out 155457.012 | DEBUG | prefect.events.clients - Reconnecting... 155457.052 | DEBUG | prefect.events.clients - pinging... 155457.054 | DEBUG | prefect.events.clients - authenticating... 155457.056 | DEBUG | prefect.events.clients - auth result {'type': 'auth_success'} 155457.057 | DEBUG | prefect.events.clients - filtering events since 2025-08-07 155357.056985+00:00... 155457.059 | DEBUG | prefect.client - Connecting to API at http://prefect.qa.gcp.radicant.com/api/ 155457.059 | DEBUG | prefect.events.clients - Reconnecting websocket connection. 155457.059 | DEBUG | prefect.events.clients - Opening websocket connection. 155457.085 | DEBUG | prefect.events.clients - Pinging to ensure websocket connected. 155457.086 | DEBUG | prefect.events.clients - Pong received. Websocket connected. 155457.087 | DEBUG | prefect.events.clients - Resending 0 unconfirmed events. 155457.087 | DEBUG | prefect.events.clients - Finished resending unconfirmed events. 155457.088 | DEBUG | prefect.runner - Limit slot acquired for flow run '4f983d8f-795b-4d96-86b9-21d7d0e651de' 155457.108 | DEBUG | prefect.runner - Stopping runner... 155457.111 | DEBUG | prefect.events.clients - Connection closed with "OK" status + Exception Group Traceback (most recent call last): | File "/usr/local/lib/python3.12/site-packages/prefect/cli/_utilities.py", line 44, in wrapper | return fn(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/cli/_types.py", line 156, in sync_fn | return asyncio.run(async_fn(*args, **kwargs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/runners.py", line 195, in run | return runner.run(main) | ^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run | return self._loop.run_until_complete(task) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/asyncio/base_events.py", line 691, in run_until_complete | return future.result() | ^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/cli/flow_run.py", line 412, in execute | await runner.execute_flow_run(id) | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 571, in execute_flow_run | async with context: | ^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 1536, in aexit | await self._exit_stack.__aexit__(*exc_info) | File "/usr/local/lib/python3.12/contextlib.py", line 754, in aexit | raise exc_details[1] | File "/usr/local/lib/python3.12/contextlib.py", line 737, in aexit | cb_suppress = await cb(*exc_details) | ^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
m
✅ thought for 17.4 seconds
d
@Marvin part 2 of the logs +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/usr/local/lib/python3.12/site-packages/prefect/runner/runner.py", line 576, in execute_flow_run | flow_run = await self._client.read_flow_run(flow_run_id) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/_flow_runs/client.py", line 668, in read_flow_run | response = await self.request( | ^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request | return await self._client.send(request) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 379, in send | response.raise_for_status() | File "/usr/local/lib/python3.12/site-packages/prefect/client/base.py", line 163, in raise_for_status | raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.cause | prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url 'http://my-prefect-url/api/flow_runs/4f983d8f-795b-4d96-86b9-21d7d0e651de' | Response: {'exception_message': 'Unauthorized'} | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401 +------------------------------------ An exception occurred.
m
✅ thought for 17.1 seconds
While we wait for the second part of the logs, could you verify a few things: 1. Check the exact value of your auth string in the pod (with sensitive info redacted):
Copy code
bash
kubectl exec -it <pod-name> -n prefect -- env | grep PREFECT_API_AUTH_STRING
2. Verify there are no extra characters or spaces in the URL:
Copy code
bash
kubectl exec -it <pod-name> -n prefect -- env | grep PREFECT_API_URL
3. Also, since you're using a custom domain (prefect.qa.gcp.radicant.com), can you confirm: - Is the API endpoint using HTTP or HTTPS? - Are there any proxy or ingress configurations in between? - Can you curl the API endpoint directly from within the pod? Try this test from within a pod:
Copy code
bash
kubectl run curl-test --image=curlimages/curl -i --rm --restart=Never -- \
  curl -v -H "Authorization: Basic $(echo -n 'your-username:your-password' | base64)" \
  <http://prefect.qa.gcp.radicant.com/api/hello>
Please share: 1. The second part of the logs 2. The results of checking the environment variables 3. The result of the curl test This will help us determine if it's a configuration issue or potentially a networking/proxy issue.
Looking at the error logs, I notice a few important things: 1. You're using Python 3.12 and what appears to be Prefect 3.x (based on the import paths) 2. The error is occurring when trying to read the flow run information 3. The URL format in the error message has angle brackets (
<http://...>
), which shouldn't be there Let's fix this step by step: 1. First, let's verify the environment variables in the pod:
Copy code
bash
kubectl exec -it <your-pod-name> -n prefect -- env | grep PREFECT
2. Check your server configuration. On your Prefect server, verify:
Copy code
bash
echo $PREFECT_SERVER_API_AUTH_STRING
3. Update your base job template with these specific changes:
Copy code
json
{
  "variables": {
    "type": "object",
    "properties": {
      "env": {
        "type": "object",
        "title": "Environment Variables",
        "default": {
          "PREFECT_API_URL": "<http://prefect.qa.gcp.radicant.com/api>"  // Remove any <> brackets!
        }
      }
    }
  },
  "job_configuration": {
    "job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "containers": [
              {
                "name": "prefect-job",
                "env": [
                  {
                    "name": "PREFECT_API_URL",
                    "value": "<http://prefect.qa.gcp.radicant.com/api>"
                  },
                  {
                    "name": "PREFECT_SERVER_API_AUTH_STRING",  // Add this
                    "valueFrom": {
                      "secretKeyRef": {
                        "name": "prefect-api-auth",
                        "key": "auth_string"
                      }
                    }
                  },
                  {
                    "name": "PREFECT_API_AUTH_STRING",  // Keep this
                    "valueFrom": {
                      "secretKeyRef": {
                        "name": "prefect-api-auth",
                        "key": "auth_string"
                      }
                    }
                  }
                ]
              }
            ]
          }
        }
      }
    }
  }
}
4. Update your Kubernetes secret to ensure the auth string is properly formatted:
Copy code
bash
kubectl create secret generic prefect-api-auth \
  --namespace prefect \
  --from-literal=auth_string='username:password' \
  --dry-run=client -o yaml | kubectl apply -f -
5. Quick test to verify the authentication:
Copy code
bash
# Create a test pod
kubectl run prefect-test --image=prefecthq/prefect:3-latest -it --rm \
  --env="PREFECT_API_URL=<http://prefect.qa.gcp.radicant.com/api>" \
  --env="PREFECT_API_AUTH_STRING=username:password" \
  --env="PREFECT_SERVER_API_AUTH_STRING=username:password" \
  -- python3 -c "
import asyncio
from prefect.client import get_client
async def test():
    client = get_client()
    try:
        await client.hello()
        print('Authentication successful!')
    except Exception as e:
        print(f'Authentication failed: {e}')
asyncio.run(test())
"
Key changes: 1. Added 
PREFECT_SERVER_API_AUTH_STRING
alongside 
PREFECT_API_AUTH_STRING
2. Removed any angle brackets from the API URL 3. Ensured the auth string is in the exact format 
username:password
Could you please: 1. Confirm if your API URL in the configuration actually has angle brackets (they need to be removed if present) 2. Try adding both auth string environment variables as shown above 3. Run the test pod command to verify authentication works directly 4. Share the output of 
kubectl exec
showing the environment variables This will help us identify exactly where the authentication is breaking down.
✅ thought for 50.9 seconds
Based on this research and the error you're seeing, here's what I recommend: 1. Update your base job template to ensure the URL is properly formatted without angle brackets:
Copy code
json
{
  "variables": {
    "type": "object",
    "properties": {
      "env": {
        "type": "object",
        "title": "Environment Variables",
        "default": {
          "PREFECT_API_URL": "<http://my-prefect-url/api>"  // No angle brackets!
        }
      }
    }
  }
}
2. Double-check your authentication string format. It should be exactly:
Copy code
username:password
No spaces, no special characters, no angle brackets. Could you please: 1. Run the environment variable check command I provided above 2. Try the test pod command 3. Confirm if your base job template has any angle brackets in the URL This will help us identify exactly where the formatting issue is occurring.
Open in Slack
PreviousNext
Powered by