https://prefect.io
Join Slack
Hello Guys. I need some clarification here. When I have a docker worker that is started through `pre...
# ask-community
a
Hello Guys. I need some clarification here. When I have a docker worker that is started through 
prefect worker start -t docker --pool <pool-name>
and it pulls an image that i specified in a deployment.. How can I ensure that the container is ran with nvidia-runtime? This is so the docker container has access to my host's nvidia driver....
k
You're looking for something like the equivalent to
Copy code
docker run --gpus all
?
or maybe more accurately: 
docker run --runtime=nvidia --gpus all
a
The 
runtime=nvidia
, because it seems to me the worker cannot actually detect my host gpu when the worker runs the container
I have the following variables in my docker image:
Copy code
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES all
so that takes care of the --gpus all
k
it looks like we're not accounting for 
runtime
on the worker, but it should be possible with a few changes
a
what kind of changes? I am not sure if this is what's expected, but it seems that the container running inside the worker cannot see the underlying host nvidia driver/toolkit because of this?
@Kevin Grismore
k
sorry I was on a call
I think this is because of how we're actually starting the container
a
can't i just set the default run-time to be nvidia?
Copy code
{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}
k
yeah, is that something you've tried?
a
not sure if it would work, but I thought maybe I can set it here: /etc/docker/daemon.json
k
we create the container through the docker python client, where 
runtime
is a kwarg but it doesn't look like we're accounting for it when we build the config that makes up those kwargs
but setting it as the default runtime might work too
a
I am trying it now!
That did not work 😓
k
dang
a
what are your thoughts on using a kubernetes worker?
k
people are definitely using the k8s worker and doing GPU stuff
16 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by