https://prefect.io logo
Join Slack
Powered by
Hello Guys. I need some clarification here. When I...
# ask-community
a
Hello Guys. I need some clarification here. When I have a docker worker that is started through 
prefect worker start -t docker --pool <pool-name>
and it pulls an image that i specified in a deployment.. How can I ensure that the container is ran with nvidia-runtime? This is so the docker container has access to my host's nvidia driver....
k
You're looking for something like the equivalent to
Copy code
docker run --gpus all
?
or maybe more accurately: 
docker run --runtime=nvidia --gpus all
a
The 
runtime=nvidia
, because it seems to me the worker cannot actually detect my host gpu when the worker runs the container
I have the following variables in my docker image:
Copy code
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES all
so that takes care of the --gpus all
k
it looks like we're not accounting for 
runtime
on the worker, but it should be possible with a few changes
a
what kind of changes? I am not sure if this is what's expected, but it seems that the container running inside the worker cannot see the underlying host nvidia driver/toolkit because of this?
@Kevin Grismore
k
sorry I was on a call
I think this is because of how we're actually starting the container
a
can't i just set the default run-time to be nvidia?
Copy code
{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}
k
yeah, is that something you've tried?
a
not sure if it would work, but I thought maybe I can set it here: /etc/docker/daemon.json
k
we create the container through the docker python client, where 
runtime
is a kwarg but it doesn't look like we're accounting for it when we build the config that makes up those kwargs
but setting it as the default runtime might work too
a
I am trying it now!
That did not work 😓
k
dang
a
what are your thoughts on using a kubernetes worker?
k
people are definitely using the k8s worker and doing GPU stuff
15 Views
Open in Slack
PreviousNext
Powered by