Infrastructure Pending

usually means Prefect created or is tracking the run’s infrastructure, but Kubernetes has not gotten the pod to a usable/running state yet. In a self-hosted Prefect + K8s setup, the most common causes are: 1. The pod is unschedulable - not enough CPU / memory in the cluster - node selectors / affinity rules don’t match any nodes - taints exist but your pod has no tolerations - namespace quotas / limit ranges block the pod Check:

bash
kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Look for messages like: -

FailedScheduling

-

0/3 nodes are available

-

Insufficient cpu

-

Insufficient memory

2. Image pull problems - bad image name/tag - private registry auth missing -

imagePullSecrets

not configured - registry/network issues Check:

bash
kubectl describe pod <pod-name> -n <namespace>

Look for: -

ImagePullBackOff

-

ErrImagePull

3. Worker / work pool job template issues If your Kubernetes work pool is creating jobs with a bad namespace, service account, image, resource request, or pod spec, the run can sit there waiting forever. Useful check:

bash
prefect work-pool inspect <pool-name>

4. RBAC / service account problems The worker may not have permission to create or watch jobs/pods in the target namespace. Check:

bash
kubectl auth can-i create jobs --as=system:serviceaccount:<ns>:<sa> -n <ns>
kubectl auth can-i get pods --as=system:serviceaccount:<ns>:<sa> -n <ns>

5. Namespace quota / policy restrictions Even if the pod spec is valid, ResourceQuota, LimitRange, admission controllers, or network policies can prevent startup. Check:

bash
kubectl describe resourcequota -n <namespace>
kubectl describe limitrange -n <namespace>

6. Init container / admission webhook delays Sometimes the pod is technically still pending because init containers never finish, or a webhook is delaying admission. Check:

bash
kubectl describe pod <pod-name> -n <namespace>

7. Prefect’s Kubernetes observer / state updates are not healthy In Prefect 3, Kubernetes integration tracks pod state and proposes state transitions like

Infrastructure Pending

. If that observer or API communication is broken, the UI state can lag or appear stuck even if the pod changed state. So also check: - worker logs - Prefect server/API reachability from the cluster - whether the pod is actually still

Pending

in Kubernetes, or only in Prefect UI Most important first step Find one stuck run, then map it to its Kubernetes pod/job and run:

bash
kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -50

That usually tells you the reason immediately. What this state means internally In Prefect 3.x,

Infrastructure Pending

is a named state under

PENDING

used when the infrastructure exists but has not started running the flow yet. For Kubernetes, that often corresponds to the pod being in K8s

Pending

phase. Practical checklist - Is the pod actually created? - Is it still

Pending

in Kubernetes? - Any

FailedScheduling

events? - Any

ImagePullBackOff

/

ErrImagePull

? - Is the worker using the right namespace/service account? - Are resource requests too high? - Are quotas or taints blocking it? If you want, send me: - your work pool type/config - one

kubectl describe pod ...

output from a stuck run - worker logs for that run and I can help pinpoint the exact cause. Relevant references: - Prefect docs - Workers and work pools - Prefect Kubernetes integration