<@ULVA73B9P> If I have multiple flow runs in state...
# ask-marvin
r
@Marvin If I have multiple flow runs in state InfrastructurePending because of node ressources insufficent. When ressources are available how is going to be selected the flow run that will run ?
m
thought for 130.0 seconds
When a flow run is already in
InfrastructurePending
, it has already been selected by a worker. So the key point is: Prefect does not re-pick among
InfrastructurePending
runs when resources free up.
Instead: - a worker has already submitted infrastructure for that run - the run stays attached to that infrastructure attempt - when the underlying platform can finally start it (for example Kubernetes finds node capacity), that same run proceeds to
Running
So if you have several runs in
InfrastructurePending
because the cluster has insufficient resources: - Prefect itself is not doing a fresh scheduling choice at that moment - the underlying infrastructure/platform decides which pending workload actually gets capacity first - in Kubernetes, for example, that depends on Kubernetes scheduling behavior, pod priority/preemption, resource requests, taints/tolerations, affinity rules, etc. What does Prefect choose? Prefect chooses when taking runs from
Scheduled
into execution. That selection is based on: 1. Work queue priority Lower priority number = higher priority 2. Scheduled start time Earlier
next_scheduled_start_time
goes first within the same queue priority Also, runs in `Pending`/`InfrastructurePending` count against work pool / queue concurrency limits, so they can block additional runs from being claimed. So the lifecycle is roughly:
Copy code
text
Scheduled
  -> selected by worker according to queue priority + scheduled time
  -> Pending / InfrastructurePending
  -> waits for infra platform
  -> Running when platform actually starts it
In your case, if the bottleneck is node resources insufficient: - Prefect selected those runs earlier - Kubernetes/infrastructure decides which one actually starts first once capacity is available If you want, I can also explain this specifically for: - Kubernetes workers - work pool / queue priority behavior - or how to avoid many runs getting stuck in
InfrastructurePending
in the first place.