<@ULVA73B9P> I am using Kubernetes worker type, bu...
# ask-marvin
b
@Marvin I am using Kubernetes worker type, but the worker fails to submit to infrastructure. I am using the
3.6.26-python3.13-kubernetes
image.
Copy code
Failed to submit flow run to infrastructure: ClientConnectorCertificateError: Cannot connect to host XXX.XX.X.X:XXX ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)')]
m
thought for 136.8 seconds
This error is usually not a Prefect bug — it’s the Kubernetes client inside the worker failing TLS validation when talking to your Kubernetes API server. The key part is:
Copy code
SSLCertVerificationError: ... Missing Authority Key Identifier
That generally points to a problem with the API server certificate chain / CA data that the worker is using, especially with newer Python/OpenSSL behavior in the
3.6.26-python3.13-kubernetes
image. What’s happening - Prefect’s Kubernetes worker uses the Kubernetes async client to submit jobs/pods - That client validates the cluster’s TLS cert using your kubeconfig or in-cluster config - TLS verification is failing before Prefect can create the infrastructure Most likely causes 1. Stale or incorrect kubeconfig being provided to the worker 2. Broken/incomplete CA chain for the Kubernetes API server 3. A certificate that older runtimes tolerated, but Python 3.13/OpenSSL now rejects What to try first 1. Regenerate your kubeconfig / cluster credentials - EKS:
Copy code
bash
     aws eks update-kubeconfig --name <cluster-name> --region <region>
- GKE:
Copy code
bash
     gcloud container clusters get-credentials <cluster-name>
- AKS:
Copy code
bash
     az aks get-credentials --resource-group <rg> --name <cluster-name>
2. Check whether the worker is using in-cluster config or an external kubeconfig - If your worker is running inside Kubernetes, letting it use in-cluster auth is often the cleanest path - If you are passing a
cluster_config
, verify that kubeconfig’s
certificate-authority-data
is current 3. Inspect the CA certificate in the kubeconfig You want to confirm the CA/cert chain is valid and complete:
Copy code
bash
   kubectl config view --raw
If you want to inspect the cert:
Copy code
bash
   kubectl config view --raw | grep certificate-authority-data | sed 's/.*: //' | base64 -d | openssl x509 -text -noout
4. Test from the same environment/image as the worker If possible, shell into a container based on
3.6.26-python3.13-kubernetes
and test
kubectl
/ API access there. That helps confirm it’s a runtime/cert issue rather than Prefect config. Important Prefect-specific note Prefect’s Kubernetes worker does not expose a “skip TLS verification” style setting for this path. It relies on the Kubernetes client configuration being valid. So the real fix is usually: - update kubeconfig - fix the cluster/API server cert chain - or use in-cluster auth instead of a problematic external kubeconfig Things I would avoid - Disabling SSL verification globally unless this is just a temporary debugging step - Assuming this is caused by your deployment YAML itself Useful docs - Prefect Kubernetes worker docs: Kubernetes worker docs - Prefect work pools/workers overview: Work pools and workers If you want, paste: - whether the worker is running inside the cluster or outside it - whether you’re using a
cluster_config
/ kubeconfig block - your cloud provider or cluster type (
EKS
,
GKE
,
AKS
, on-prem) and I can help narrow down the exact fix.
b
@Marvin Would downgrading to python3.12 version of the image fix the issue?
m
thought for 9.5 seconds
It might, but I wouldn’t treat it as the real fix. Why: - The error strongly suggests your cluster cert/CA chain is a bit non-standard - Python 3.13 / newer OpenSSL may be stricter and reject certs that Python 3.12 happened to accept - So switching to the
python3.12-kubernetes
image could work as a workaround, but it would mostly confirm a certificate compatibility problem rather than solve it Recommendation - Yes, it’s reasonable to test the Python 3.12 image as a quick diagnostic - But if it works, I’d still plan to: - refresh/regenerate kubeconfig - verify the API server CA/cert chain - prefer in-cluster auth if applicable How to interpret the result - 3.12 works, 3.13 fails → very likely stricter TLS validation in the newer runtime - Both fail → almost certainly a kubeconfig / cluster certificate issue - Both work intermittently → less likely a pure cert issue; worth checking networking/proxy setup too If you do test it, try to keep everything else the same except the image tag so the comparison is clean. If you want, I can also help you identify the exact Python 3.12 Kubernetes image tag to try for your Prefect version.
b
@Marvin The solution was to downgrade to Python 3.12 👍
m
thought for 8.8 seconds
Nice — that’s a useful data point. If downgrading to Python 3.12 fixed it, that strongly suggests a Python/OpenSSL certificate validation compatibility issue with your cluster cert chain, rather than a Prefect configuration problem. So the practical takeaway is: - Immediate workaround: use the Python 3.12 Kubernetes image - Underlying cause: your Kubernetes API server cert / CA chain is likely accepted by 3.12 but rejected by 3.13’s stricter SSL validation If you want to make this more permanent, I’d still recommend eventually checking: - the kubeconfig
certificate-authority-data
- whether the API server cert chain is complete - whether your cluster/provider has a certificate rotation or kubeconfig refresh command available If helpful, I can also help you turn this into: 1. a short internal root-cause summary for your team, or 2. a GitHub issue/discussion-worthy repro note in case others hit the same
python3.13-kubernetes
image behavior.