<@ULVA73B9P> how setup that job which spin up by worker run only on on-demand nodes <https://github....
d
m
thought for 342.8 seconds
Great question. You want the flow-run Kubernetes Jobs created by the Prefect worker to land only on on-demand nodes. You’ll do this by setting node scheduling fields (nodeSelector/affinity/tolerations) in the worker’s base job template in the Helm values. High level: - Label your on-demand nodes (or use cloud-provided labels) - EKS:
<http://eks.amazonaws.com/capacityType=ON_DEMAND|eks.amazonaws.com/capacityType=ON_DEMAND>
(Spot is
SPOT
) - GKE: preemptible nodes have
<http://cloud.google.com/gke-preemptible=true|cloud.google.com/gke-preemptible=true>
(so select nodes where it’s not true) - Or add your own label, e.g.,
capacity=on-demand
- Configure the worker’s base job template so every flow-run Job/Pod uses those scheduling constraints - Ensure you do not add tolerations for any taints that only exist on spot nodes Example values.yaml snippets you can drop under the worker’s work pool base job template: Option A: Simple nodeSelector (EKS example)
Copy code
worker:
  workPool:
    type: kubernetes
    name: my-k8s-pool
    baseJobTemplate:
      pod:
        spec:
          # Force flow-run pods onto on-demand nodes
          nodeSelector:
            <http://eks.amazonaws.com/capacityType|eks.amazonaws.com/capacityType>: "ON_DEMAND"
          # Ensure we don't tolerate any spot-only taints
          tolerations: []
Option B: Node affinity (more explicit)
Copy code
worker:
  workPool:
    type: kubernetes
    name: my-k8s-pool
    baseJobTemplate:
      pod:
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: <http://eks.amazonaws.com/capacityType|eks.amazonaws.com/capacityType>
                    operator: In
                    values: ["ON_DEMAND"]
          tolerations: []
Option C: Custom label you control (works across providers)
Copy code
worker:
  workPool:
    type: kubernetes
    name: my-k8s-pool
    baseJobTemplate:
      pod:
        spec:
          nodeSelector:
            capacity: "on-demand"
          tolerations: []
Notes: - These keys live under the “base job template” for the Kubernetes worker (the section you linked is exactly where you put them). This controls the flow-run pods created by the worker, not the worker pod itself. - If your spot node group is tainted (e.g.,
spotInstance=true:NoSchedule
), make sure you do not add a matching toleration in the base job template — otherwise pods could still land there. - After updating values, upgrade the release:
Copy code
helm upgrade <release-name> prefect/prefect-worker -f values.yaml
If you share: - Your Kubernetes provider (EKS/GKE/AKS/other) - The label/taint you use to distinguish on-demand vs spot …I can tailor the exact snippet for your cluster. Helpful links: - Prefect Helm worker values (the section you referenced): values.yaml lines 113–128 - Prefect 3.x Kubernetes deployment guidance: https://docs-3.prefect.io/v3/how-to-guides/deployment_infra/kubernetes