https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I am seeing intermittent error "kuber...
# ask-marvin
u
@Marvin I am seeing intermittent error "kubernetes.client.exceptions.ApiException: (500)" from flow and agent containers in prefect v2. what could be the probable cause
Copy code
ERROR:agent:Exception encountered while deploying flow run a79c877a-e321-4aab-b571-d3a9f7c49a60
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 391, in _deploy_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 431, in deploy_flow
    namespace=self.namespace, body=job_spec
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 66, in create_namespaced_job
    return self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api/batch_v1_api.py", line 175, in create_namespaced_job_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
    _preload_content, _request_timeout, _host)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 397, in request
    body=body)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 280, in POST
    body=body)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 233, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (500)
m
✅ thought for 100.8 seconds
Short answer: a 500 from the Kubernetes API when your agent tries to create a Job is almost always a cluster-side issue (admission webhooks or API server/etcd health), not a Prefect configuration bug. A few notes based on your traceback - The stack trace shows 
prefect/agent/kubernetes/agent.py
running on Python 3.7. That looks like a Prefect 1.x Kubernetes Agent, not Prefect 2.x (Prefect 2+ requires Python >= 3.8 and uses workers instead of agents). Can you share the output of 
prefect version
to confirm? - Regardless of version, the failure is happening inside the Kubernetes API when handling Job creation. Most common root causes for intermittent 500s on Job creation 1) Admission webhooks intermittently failing - Examples: Istio/Linkerd sidecar injector, Kyverno, OPA/Gatekeeper, Datadog, custom org webhooks. - Symptoms: 500 with a message like “Internal error occurred: failed calling webhook …”, timeouts, TLS issues (expired cert), or webhook pod not ready. - Docs: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ 2) API server / etcd under load or unstable - Symptoms: 500 with messages like “etcdserver: request timed out”, “context deadline exceeded”, spikes in apiserver 5xx. - Typically correlates with control-plane load, etcd health, or networking blips. 3) Misbehaving controllers or aggregator - Less common, but a controller or aggregated API can error during admission, returning 5xx. 4) Very large objects or unusual defaults causing server-side panics - Rare, but can happen if the Job spec is unusually large (massive env/annotations) or triggers a bug in a webhook. What to check next (to turn the 500 into a specific cause) - Capture the full ApiException body - The Python client’s 
ApiException
has 
e.status
, 
e.reason
, and 
e.body
which usually contains the real clue. Your logs only show “(500)”. If possible, enable debug logs on the agent and ensure the exception body is printed. - For Prefect 1.x agents, set 
PREFECT__LOGGING__LEVEL=DEBUG
. For Prefect 2+/3, use 
PREFECT_LOGGING_LEVEL=DEBUG
. - Look for cluster events and apiserver/webhook errors - Events:
Copy code
kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -n 100
- API server logs (managed clusters surface these in provider logs): - Look for “failed calling webhook”, “etcdserver”, “timeout”, “panic”. - Webhooks health:
Copy code
kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
    kubectl get pods -A -l app=istiod,app=gatekeeper,app=kyverno,admission-webhook=true
Check for restarts, OOMs, or pending pods. - Quick baseline test (outside Prefect) to see if Job creation itself is flaky:
Copy code
kubectl -n <namespace> create job job-smoke --image=busybox -- sleep 1
If that intermittently fails, it’s definitely cluster-side. - RBAC sanity check (even though RBAC issues are usually 403, not 500):
Copy code
kubectl auth can-i create jobs --as=system:serviceaccount:<namespace>:<serviceaccount> -n <namespace>
Mitigations once you identify the culprit - If a webhook is failing: - Fix webhook health (HA replicas, resource requests/limits, autoscaling). - Verify webhook TLS cert validity and CA bundle. - Adjust timeouts/failurePolicy (e.g., 
Ignore
) or use namespaceSelector/objectSelector to exclude the Prefect namespace if appropriate. - If API server/etcd is overloaded: - Reduce control-plane pressure (fewer large writes/bursty controllers), scale/upgrade the control plane, or investigate network stability. - If the Job spec is huge: - Trim annotations/env vars to shrink the object size. - Add client-side retries on job creation: - Intermittent 500s often succeed on retry. If you’re on Prefect 1.x agents, this may require wrapping the call or moving to modern workers. Recommendation on Prefect version
- If you truly are on Prefect 1.x (as the trace suggests), it’s EOL. We recommend upgrading to Prefect 3.x (or at least 2.x) and using Kubernetes workers, which have better resiliency and observability: https://docs.prefect.io/latest/deploy/infrastructure/kubernetes/ Could you share: - Output of 
prefect version
and Python version in the agent/worker container - Kubernetes version (
kubectl version --short
) and platform (EKS/GKE/AKS/on-prem) - Any admission controllers you’re running (Istio/Linkerd, Kyverno, Gatekeeper, Datadog, custom) - The full 
ApiException
body if you can capture it With that info (especially the exception body), I can help pinpoint the exact cause and fix.
5 Views
Open in Slack
PreviousNext
Powered by