Hello @Marvin I have a kubernetes worker installed through helm on my eks cluster and I can accept jobs submitted through the dashboard into that. My question is, I want to schedule flows using that worker, however, I want to specify machine types on my cluster for the container to run on. Looking through the docs, I see that I can override the default base job template under job configuration. How do i specifically tell kubernetes to use a specific machine from a node group i have on my cluster?

It appears that the Prefect documentation provides some insights into the configuration of Kubernetes jobs, but none of these excerpts directly answer your question. To specify a machine type from a node group in your EKS cluster for a Kubernetes job, you would generally use node selectors, tolerations, or affinity settings in your Kubernetes job specification. To customize the Kubernetes job used by Prefect for running flows, you would modify the default base job template to include the necessary specifications. You might typically do this by setting the

nodeSelector

,

affinity

, or

tolerations

properties in the job's pod template. Unfortunately, the excerpts provided don't contain the specific details on how to achieve this within the Prefect 2.x context. However, to get a more pointed answer, let's search again with a more targeted question that explicitly asks about setting node selectors or affinity in the job template for Prefect Kubernetes jobs. It seems that my search was not as fruitful as I had hoped; the excerpts from Prefect documentation do not address the specific question of setting node selectors or affinity in the Kubernetes job template for executing flows with Prefect in your EKS cluster. Given the lack of specific documentation on this subject, a common Kubernetes approach is to add fields such as

nodeSelector

,

affinity

, and

tolerations

to your job specification to tell Kubernetes which nodes should be used for running the job. In general, you would do something like this in your Kubernetes job YAML:

yaml
spec:
  template:
    spec:
      nodeSelector:
        key: value
      affinity:
        nodeAffinity:
          ...
      tolerations:
        ...

Replace

key: value

with the proper label that corresponds to your desired node group. Configure the

affinity

and

tolerations

as required by your setup. However, the way to incorporate such settings in Prefect's job configuration might require extracting and modifying the default job template YAML, then using it in your Prefect deployment setup. Such modifications would typically be conducted on the deployment level, and the exact process might be clarified within Prefect's CLI or job configuration options. For detailed guidance, you may need to reach out to the Prefect community or Prefect support, or explore Prefect's CLI documentation further to uncover the nuances of integrating these Kubernetes-specific settings into your flows. You can start with the Prefect Kubernetes tutorial, and if needed, look for further information or raise an issue in the Prefect GitHub repository.

I'd refer to that advanced settings tip in the blue box under this header. basically you'll want to end up with a

...
        "nodeSelector": "{{ node_selector }}"

in the right spot in your template, along with a

"node_selector": {
  "title": "Node Selector",
  "type": "object"
},

variable in your variables, both part of the same json under the advanced tab. once that's saved, you can add a

node_selector

job variable to your deployments

Thanks kevin!!

if you're comfortable with writing k8s job manifests, I'd recommend grabbing the job_manifest part of the json, dropping it into a json->yaml converter, making your changes, and then converting it back to json. helps avoid formatting errors, missing commas and quotes, etc

I was just looking at that in the docs. I assume there is more to do if I am asking for a gpu from one of the node groups in my cluster!

yeah for sure