Denys Y
09/10/2025, 8:07 AMMarvin
09/10/2025, 8:07 AMMarvin
09/10/2025, 8:13 AM<http://eks.amazonaws.com/capacityType=ON_DEMAND|eks.amazonaws.com/capacityType=ON_DEMAND> (Spot is SPOT)
- GKE: preemptible nodes have <http://cloud.google.com/gke-preemptible=true|cloud.google.com/gke-preemptible=true> (so select nodes where it’s not true)
- Or add your own label, e.g., capacity=on-demand
- Configure the worker’s base job template so every flow-run Job/Pod uses those scheduling constraints
- Ensure you do not add tolerations for any taints that only exist on spot nodes
Example values.yaml snippets you can drop under the worker’s work pool base job template:
Option A: Simple nodeSelector (EKS example)
worker:
workPool:
type: kubernetes
name: my-k8s-pool
baseJobTemplate:
pod:
spec:
# Force flow-run pods onto on-demand nodes
nodeSelector:
<http://eks.amazonaws.com/capacityType|eks.amazonaws.com/capacityType>: "ON_DEMAND"
# Ensure we don't tolerate any spot-only taints
tolerations: []
Option B: Node affinity (more explicit)
worker:
workPool:
type: kubernetes
name: my-k8s-pool
baseJobTemplate:
pod:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: <http://eks.amazonaws.com/capacityType|eks.amazonaws.com/capacityType>
operator: In
values: ["ON_DEMAND"]
tolerations: []
Option C: Custom label you control (works across providers)
worker:
workPool:
type: kubernetes
name: my-k8s-pool
baseJobTemplate:
pod:
spec:
nodeSelector:
capacity: "on-demand"
tolerations: []
Notes:
- These keys live under the “base job template” for the Kubernetes worker (the section you linked is exactly where you put them). This controls the flow-run pods created by the worker, not the worker pod itself.
- If your spot node group is tainted (e.g., spotInstance=true:NoSchedule), make sure you do not add a matching toleration in the base job template — otherwise pods could still land there.
- After updating values, upgrade the release:
helm upgrade <release-name> prefect/prefect-worker -f values.yaml
If you share:
- Your Kubernetes provider (EKS/GKE/AKS/other)
- The label/taint you use to distinguish on-demand vs spot
…I can tailor the exact snippet for your cluster.
Helpful links:
- Prefect Helm worker values (the section you referenced): values.yaml lines 113–128
- Prefect 3.x Kubernetes deployment guidance: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/kubernetesBring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by