Hi, I’m trying to get different subflows running on pods with different resources, like high memory for one and GPUs for another. It seems like creating different worker pools might be the way to go, but I’m having issues setting up other worker pools. I tried to create one for GPUs, and started out simple by basically just duplicating my default pool as a starting point. I then realized I don’t think the
helm install
will work if its within the same namespace, so I tried running it in a new namespace, but its just showing
Error from server (BadRequest): container "prefect-worker" in pod "prefect-worker-8678675458-kznph" is waiting to start: CreateContainerConfigError
in the worker pod logs.
Any advice on the general setup for multiple worker pools inside k8s?
j
Jamie Zieziula
12/05/2023, 4:52 PM
hi Andrew! you should be able to deploy more than one worker into a single namespace. it’s documented in the worker readme - just pass in a value to fullnameoverride!
a
Andrew
12/05/2023, 5:24 PM
perfect, I hadn’t seen that readme. looks like the worker is now creating, I’ll give that a try! thanks!
Andrew
12/05/2023, 5:27 PM
If I make changes to the base job template for a worker pool (in the UI, under the advanced tab), do I have to uninstall and reinstall the worker for it to use the updates?
k
Kevin Grismore
12/05/2023, 5:46 PM
nope! the worker grabs the current job template when a run of a deployment starts
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.