Romain Lefebvre
07/22/2026, 11:43 AMMarvin
07/22/2026, 11:43 AMMarvin
07/22/2026, 11:45 AMInfrastructurePending, it has already been selected by a worker.
So the key point is:
Prefect does not re-pick among InfrastructurePending runs when resources free up.
Instead:
- a worker has already submitted infrastructure for that run
- the run stays attached to that infrastructure attempt
- when the underlying platform can finally start it (for example Kubernetes finds node capacity), that same run proceeds to Running
So if you have several runs in InfrastructurePending because the cluster has insufficient resources:
- Prefect itself is not doing a fresh scheduling choice at that moment
- the underlying infrastructure/platform decides which pending workload actually gets capacity first
- in Kubernetes, for example, that depends on Kubernetes scheduling behavior, pod priority/preemption, resource requests, taints/tolerations, affinity rules, etc.
What does Prefect choose?
Prefect chooses when taking runs from Scheduled into execution. That selection is based on:
1. Work queue priority
Lower priority number = higher priority
2. Scheduled start time
Earlier next_scheduled_start_time goes first within the same queue priority
Also, runs in `Pending`/`InfrastructurePending` count against work pool / queue concurrency limits, so they can block additional runs from being claimed.
So the lifecycle is roughly:
text
Scheduled
-> selected by worker according to queue priority + scheduled time
-> Pending / InfrastructurePending
-> waits for infra platform
-> Running when platform actually starts it
In your case, if the bottleneck is node resources insufficient:
- Prefect selected those runs earlier
- Kubernetes/infrastructure decides which one actually starts first once capacity is available
If you want, I can also explain this specifically for:
- Kubernetes workers
- work pool / queue priority behavior
- or how to avoid many runs getting stuck in InfrastructurePending in the first place.