I am deploying a kube worker using the documentation found here: https://docs.prefect.io/2.11.5/guides/deployment/kubernetes/ How can I set the taint / nodeSelector values while running the helm install command? Can it be overriden in the values.yaml config file?

hi @Eric - everything you see in here can be overridden in the values.yaml

Thanks! I also realized I can just extract the values.yaml file so I checked against that

Hm the solutions posted on that thread don't show where to set toleration / nodeSelector values. Are there any examples?

the point is that you can alter the spec however you want - there's probably not many

prefect.yaml

examples of people altering the spec in the exact way you're planning on, but it shouldnt be meaningfully different from the examples in that other thread in terms of how to accomplish a spec override relevant k8s docs

In my prefect.yaml deployment file, i have this

deployments:
- name: <>
  tags: *common_tags
  schedule: null
  entrypoint: <>
  work_pool: *common_work_pool
  run_config:
    type: "kubernetes"
    job_template_path: "./kube_prefect_job_template.yaml"

And my

kube_prefect_job_template.yaml

looks like

apiVersion: batch/v1
kind: Job
metadata:
  generateName: flow-run-
spec:
  template:
    spec:
      nodeSelector:
        "kube/nodetype": "asyncjobs"
      tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "asyncjobs"
          effect: "NoSchedule"

But these values are not getting picked up by the new pods; im not sure where prefect holds these values

when you create your deployment and go to the configuration tab on the deployment of interest, do you see any of your spec changes? i am not familiar with the run_config / job_template path syntax you're using

Nope; I'm probably not setting the values using the right fields. Wondering if there are examples for this

i would think you just want

deployments:
- name: <>
  tags: *common_tags
  schedule: null
  entrypoint: <>
  work_pool: *common_work_pool
  job_variables:
     job_manifest:
        spec:
           containers:
           # whatever you want to do to the spec

I copied the spec example from the thread and I see those values in the deployment configuration tab now, but the pod is still not picking those configs up

i would want to

k describe your-pod

and see whats going on there

Seeing

Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  21s   default-scheduler  0/4 nodes are available: 2 node(s) had untolerated taint {dedicated: asyncjobs}, 2 node(s) had untolerated taint {dedicated: webapp}. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..

You can see node selectors and tolerations are not being propagated from the deployment job arguments

Hi @Eric what do you want to be affected by the taint/nodeSelectors? the

prefect worker

deployment or the

flow

pods?

The flow pods

ICYMI in that thread i linked earlier i suggest setting the spec on the advanced tab of the work pool itself, so that each deployment flow run's pod inherits it- instead of putting it in the

prefect.yaml

(which overrides it for each deployment)

can you describe the job created in the cluster and share the output here?

@Nate just making sure I'm not misreading this:

you can set resource requests on your k8s work pool as a default in the advanced tab in the UI (picture attached) or you could override those defaults for a given deployment in your prefect.yaml like

I read this, and added it to my deployment

prefect.yaml

file

for testing - can you provide the configuration as a part of the advanced tab under the workpool? like you shared above ^

"job_manifest": {
    "spec": {
      "template": {
        "spec": {
          "tolerations": [
            {
              "key": "dedicated",
              "value": "asyncjobs",
              "effect": "NoSchedule",
              "operator": "Equal"
            }
          ],
          "nodeSelector": {
            "kube/nodetype": "asyncjobs"
          }
        }
      }
    }
  }

I read this, and added it to my deployment

prefect.yaml

file

that's one option, but by putting that config in your

prefect.yaml

you're saying "i want this job spec for this specific deployment" not "all flow runs from the work pool" if you want this spec for all flow runs from this deployment, you should put it directly on the work pool

But its defined in my work pool definition:

definitions:
  tags: &common_tags
    - "feedback-insight"
    - "dev"
  work_pool: &common_work_pool
    name: $PREFECT_WORK_POOL_NAME
    job_variables:
      image: $BACKEND_IMAGE_URI
      env:
        <>
      job_manifest:
        spec:
          template:
            spec:
              tolerations:
              - key: "dedicated"
                operator: "Equal"
                value: "asyncjobs"
                effect: "NoSchedule"
              nodeSelector:
                "kube/nodetype": "asyncjobs"

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: "parse-feedback-dev"
  tags: *common_tags
  schedule: null
  entrypoint: "inari_app/processing/feedback/feedback_parse_prefect.py:parse_feedback_and_highlights"
  work_pool: *common_work_pool

that is not your work pool definition, that is an override of the default values that already exist on your work pool

Ok. But as long as I am creating flow runs using this deployment, the job_manifest should be overriden right?

this conversation is happening across 2 different threads. may i suggest we consolidate on this one