https://prefect.io logo
Title
a

Ankit

02/08/2023, 4:10 PM
How can we create pods on different instance types in k8s? Currently I have one work queue and one agent which is scheduling pods based on whatever nodegroup is available, I used the helm installation to setup the cluster. I am probably missing on some configuration
1
t

Timo Vink

02/08/2023, 4:13 PM
Is your question "how can I choose what instance type my job runs on" or "how can I ensure my pods all run on separate machines", or something else?
a

Ankit

02/08/2023, 4:14 PM
Hey @Timo Vink my question is how can I choose instance type (or node group) to run my jobs?
t

Timo Vink

02/08/2023, 4:18 PM
You probably want to end up with a
nodeSelector
on your jobs (kubernetes feature, not a prefect feature) to make sure it uses the right instance type. You do do this by using the
customizations
parameters of the
KubernetesJob
object. You can probably do this on the command-line, but much easier if you create your
Deployment
using the Python API. Example by Anna here: https://github.com/anna-geller/prefect-deployment-patterns/blob/main/blocks/infrastructure_blocks/kubernetes-job/customizations/gpu_on_gke.py
a

Ankit

02/08/2023, 4:20 PM
That's what I was looking for, thanks a lot @Timo Vink I am currently using memory limits/requests so I can use nodeselector similar to that
t

Timo Vink

02/08/2023, 4:20 PM
What the exact value of the
nodeSelector
should be depends on the details of your cluster/environment. But if you look at your existing nodes you can probably find a prepopulated label that has the instance type on it
a

Ankit

02/08/2023, 4:21 PM
right, I can just check one of the existing pods and see what exactly should I mention
1