How can we create pods on different instance types in k8s? Currently I have one work queue and one agent which is scheduling pods based on whatever nodegroup is available, I used the helm installation to setup the cluster. I am probably missing on some configuration

Is your question "how can I choose what instance type my job runs on" or "how can I ensure my pods all run on separate machines", or something else?

Hey @Timo Vink my question is how can I choose instance type (or node group) to run my jobs?

You probably want to end up with a

nodeSelector

on your jobs (kubernetes feature, not a prefect feature) to make sure it uses the right instance type. You do do this by using the

customizations

parameters of the

KubernetesJob

object. You can probably do this on the command-line, but much easier if you create your

Deployment

using the Python API. Example by Anna here: https://github.com/anna-geller/prefect-deployment-patterns/blob/main/blocks/infrastructure_blocks/kubernetes-job/customizations/gpu_on_gke.py

That's what I was looking for, thanks a lot @Timo Vink I am currently using memory limits/requests so I can use nodeselector similar to that

What the exact value of the

nodeSelector

should be depends on the details of your cluster/environment. But if you look at your existing nodes you can probably find a prepopulated label that has the instance type on it

right, I can just check one of the existing pods and see what exactly should I mention