Thanks for the details — an httpcore.ConnectTimeout after ~10 minutes generally means the pod cannot open a TCP connection to your Prefect API at all. The ~10m duration matches Prefect’s defaults in 3.x: 60s request timeout with several retries and backoff. Most common root causes in K8s: - PREFECT_API_URL is wrong or missing /api - DNS/service name not resolvable from the job’s namespace - NetworkPolicy/firewall blocking traffic to the API service - Using HTTPS with a self-signed cert but not configuring trust - Pointing to an external LB/Ingress that is not reachable from inside the cluster Quick triage checklist 1) Verify the exact API URL inside the job container - It must include scheme and /api, e.g.: - http:

<http://prefect-api.prefect.svc.cluster.local:4200/api>

- https:

<https://prefect.mycompany.com/api>

- From the job’s container, run:

env | grep PREFECT_API_URL

2) Probe the health endpoint from the same namespace - If PREFECT_API_URL already ends with /api, you can probe “/health” off it:

kubectl run -it --rm netcheck --image=curlimages/curl --restart=Never -- \
  sh -lc 'curl -vk $PREFECT_API_URL/health'

- Or hardcode the in-cluster DNS:

kubectl run -it --rm netcheck --image=curlimages/curl --restart=Never -- \
  sh -lc 'curl -vk <http://prefect-api.prefect.svc.cluster.local:4200/api/health>'

If this cannot connect, it’s a networking/DNS/service issue, not Prefect. 3) Confirm service/endpoints and DNS

kubectl -n <your-prefect-namespace> get svc,endpoints | grep prefect
kubectl run -it --rm dnscheck --image=alpine --restart=Never -- sh -lc \
  "nslookup prefect-api.<your-prefect-namespace>.svc.cluster.local"

4) Check for NetworkPolicies

kubectl -n <your-prefect-namespace> get networkpolicies

If policies exist, ensure they allow traffic from the job’s namespace to the Prefect API service on port 4200. 5) If using HTTPS/self-signed certs - Either add your CA bundle to the container and set: -

SSL_CERT_FILE=/path/to/ca.pem

- Or for testing only: -

PREFECT_API_TLS_INSECURE_SKIP_VERIFY=true

Then test

curl -vk $PREFECT_API_URL/health

again. 6) Temporarily fail faster while debugging Set these env vars on the job so you don’t wait 10 minutes each attempt:

PREFECT_CLIENT_MAX_RETRIES=1
PREFECT_API_REQUEST_TIMEOUT=15

K8s job env example

env:
  - name: PREFECT_API_URL
    value: "<http://prefect-api.prefect.svc.cluster.local:4200/api>"
  - name: PREFECT_CLIENT_MAX_RETRIES
    value: "1"          # speeds up debugging
  - name: PREFECT_API_REQUEST_TIMEOUT
    value: "15"         # speeds up debugging
  # If using HTTPS with self-signed
  # - name: PREFECT_API_TLS_INSECURE_SKIP_VERIFY
  #   value: "true"
  # Or mount CA and set SSL_CERT_FILE

Health endpoints to use - GET /api/health: simple liveness check (no auth) - GET /api/ready: readiness (checks DB; no auth) A couple of likely gotchas to double-check - If your job runs in a different namespace than the API service, you must use the fully qualified service DNS: <service>.<namespace>.svc.cluster.local - Ensure the API service actually listens on 4200 and that your URL includes /api - If you pointed to an external ingress/hostname, confirm pods in your cluster can route to it (sometimes egress or DNS for external hosts is blocked) If you can share: - The exact

PREFECT_API_URL

you’re using - The job manifest’s env section - Whether the API service is a ClusterIP/Ingress and its namespace/service name - Whether you’re using TLS/self-signed …I can pinpoint the fix quickly.

Got it — httpx.ConnectTimeout means the TCP connection to your Prefect API never completes within the timeout window. If your URL/DNS/TLS look right, the remaining usual culprits are networking and proxies. Here’s how to pinpoint it quickly. High‑signal checks from inside the same job image - Confirm the exact URL and that Prefect is actually using it:

python - <<'PY'
import os
from prefect.settings import get_current_settings
s = get_current_settings()
print("ENV PREFECT_API_URL =", os.getenv("PREFECT_API_URL"))
print("Resolved settings api.url =", s.api.url)
print("timeout =", s.api.request_timeout, "retries =", s.client.max_retries)
PY

- Curl the health endpoint from a throwaway pod in the same namespace as the job:

kubectl run -it --rm netcheck --image=curlimages/curl --restart=Never -- \
  sh -lc 'echo $PREFECT_API_URL && curl -vk ${PREFECT_API_URL}/health'

Or hardcode the service FQDN to remove any env ambiguity:

kubectl run -it --rm netcheck --image=curlimages/curl --restart=Never -- \
  sh -lc 'curl -vk <http://prefect-api>.<namespace>.svc.cluster.local:4200/api/health'

If curl cannot connect either, it’s definitely a cluster networking/DNS/proxy issue. Common causes that still yield ConnectTimeout even when the URL is “correct” - Proxy variables in the container: If HTTP(S)_PROXY is set but internal cluster traffic shouldn’t use it, requests will route to a proxy that drops them and you’ll see ConnectTimeout. - Set NO_PROXY to include cluster domains and your service names:

env:
- name: NO_PROXY
  value: "127.0.0.1,localhost,.svc,.svc.cluster.local,.cluster.local,prefect-api,prefect-api.<namespace>.svc.cluster.local"

- Or unset HTTP_PROXY/HTTPS_PROXY in the job if they’re inherited from the base image. - Different clusters: If your job runs in a different K8s cluster than the Prefect API, a ClusterIP service like prefect-api.<namespace>.svc.cluster.local is not reachable. Use an external/ingress hostname that’s reachable from the job’s cluster instead. - NetworkPolicy/egress firewall: Policies may silently drop traffic to the API service. - Check and temporarily relax policies to allow pod-to-service on port 4200:

kubectl -n <namespace> get networkpolicies

- DNS is OK but traffic is dropped: Quick TCP probe helps confirm:

kubectl run -it --rm --image=busybox:1.36 netcat --restart=Never -- \
  sh -lc 'nc -vz prefect-api.<namespace>.svc.cluster.local 4200'

Speed up iterations while debugging

PREFECT_CLIENT_MAX_RETRIES=1
PREFECT_API_REQUEST_TIMEOUT=15

If the curl/NC tests succeed but Prefect still times out, please share: - Output of the settings print above (resolved api.url/timeout/retries) - Whether any HTTP(S)_PROXY/NO_PROXY env vars are present in the job - The exact API endpoint you can curl successfully from the pod From there, we can zero in (e.g., proxy bypass, different cluster vs. ClusterIP, or a subtle URL mismatch).

Sorry, I encountered an error while processing your request. Please try again.

If curl from the job pod times out too, a NetworkPolicy is very likely blocking traffic. You need to explicitly allow: - Egress from the job pod(s) to the Prefect API on TCP/4200 - Ingress to the Prefect API pod(s) from the job namespace - Egress from the job pod(s) to kube-dns on 53 (UDP/TCP) so DNS works Below are ready-to-use examples you can adapt. Replace: - <prefect-ns> with the namespace running the Prefect API - <jobs-ns> with the namespace running your Kubernetes Job - app: prefect-api with the actual label on your Prefect API pods (see “Find labels” below) 1) Ingress allow on the Prefect API (in <prefect-ns>)

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-jobs-to-prefect-api
  namespace: <prefect-ns>
spec:
  podSelector:
    matchLabels:
      app: prefect-api   # <-- update to match your API pod labels
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          <http://kubernetes.io/metadata.name|kubernetes.io/metadata.name>: <jobs-ns>
    ports:
    - protocol: TCP
      port: 4200

2) Egress allow from Job pods to the Prefect API and DNS (in <jobs-ns>) Option A: target API pods by namespace+label (recommended)

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: NetworkPolicy
metadata:
  name: allow-egress-to-prefect-api-and-dns
  namespace: <jobs-ns>
spec:
  podSelector: {}        # or narrow to your job pods if you have a label to match
  policyTypes:
  - Egress
  egress:
  # Allow DNS to CoreDNS/kube-dns
  - to:
    - namespaceSelector:
        matchLabels:
          <http://kubernetes.io/metadata.name|kubernetes.io/metadata.name>: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns   # or: k8s-app: coredns (depending on your cluster)
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

  # Allow Prefect API on 4200 in <prefect-ns> to pods labeled app: prefect-api
  - to:
    - namespaceSelector:
        matchLabels:
          <http://kubernetes.io/metadata.name|kubernetes.io/metadata.name>: <prefect-ns>
      podSelector:
        matchLabels:
          app: prefect-api    # <-- update to match your API pod labels
    ports:
    - protocol: TCP
      port: 4200

Option B: if your job connects via an external ingress/LB hostname instead of a ClusterIP Use ipBlock to permit egress to the LB’s IP/CIDR and the correct port (80/443 typically):

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: NetworkPolicy
metadata:
  name: allow-egress-to-prefect-ingress-and-dns
  namespace: <jobs-ns>
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  # DNS
  - to:
    - namespaceSelector:
        matchLabels:
          <http://kubernetes.io/metadata.name|kubernetes.io/metadata.name>: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns  # or coredns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

  # Egress to external LB for Prefect API
  - to:
    - ipBlock:
        cidr: 203.0.113.10/32   # <-- replace with your LB IP/CIDR
    ports:
    - protocol: TCP
      port: 443                 # or 80 if using HTTP

Find the right labels to use - Prefect API pods: - kubectl -n <prefect-ns> get pods --show-labels - Pick a stable label (e.g., app: prefect-api or app.kubernetes.io/name: prefect-api) - Job pods: - For tighter scoping, add a label to the Job’s pod template spec (e.g., job-role: prefect-flow) and use it in podSelector for the egress policy. Test after applying - Re-run a network test from the jobs namespace:

kubectl run -it --rm netcheck --image=curlimages/curl -n <jobs-ns> --restart=Never -- \
  sh -lc 'curl -vk <http://prefect-api>.<prefect-ns>.svc.cluster.local:4200/api/health'

- If you still see timeouts, describe policies to confirm they select the intended pods:

kubectl -n <prefect-ns> describe networkpolicy allow-ingress-from-jobs-to-prefect-api
kubectl -n <jobs-ns> describe networkpolicy allow-egress-to-prefect-api-and-dns

Notes

- NetworkPolicy enforcement depends on your CNI (Calico, Cilium, etc.). Ensure it supports NetworkPolicy. - If you define any egress rule for a selected pod, all other egress is denied unless explicitly allowed. Don’t forget DNS. - Adjust the port (4200) if your API listens elsewhere or you’re using an ingress on 80/443. Kubernetes NetworkPolicy docs: Kubernetes NetworkPolicies If you share: - The namespaces for the Job and Prefect API - The labels on the Prefect API pods and your Job pods - Whether you connect via ClusterIP service or external ingress …I can tailor the exact policies for your setup.