ehfd / nvidia-dind Goto Github PK

I used your image to create a container. In the container, I installed minikube. When I run minikube start, the node minikube didn't detect any GPU. I am wondering how to fix this. By the way, the command nvidia-smi works well.

docker run --gpus 1 -it --privileged --name ElasticDL -d ghcr.io/ehfd/nvidia-dind:latest
docker exec -it ElasticDL /bin/bash
# install minikube
minikube start
alias kubectl="minikube kubectl --"
kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Then the result shows that the GPU num is <none>

When I run a pod, the pod status is:

FailedScheduling
0/1 nodes are available: 1 Insufficient ndefault-schedulervidia.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

This is less a bug report and more of an FYI for anyone else trying to use this container with a pytorch or any torch?. I was able to get nvidia-smi to work inside my matroska container, but the cuda version was n/a and torch.cuda.is_available() would be False. So I changed the base from ubuntu to the cuda base that matched my AMI (11.4 in this case) and now it works!

Hi,
This is a great solution for running an isolated environment inside a docker daemon on the host machine.
Running nvidia-smi inside the dind works as expected.
However, when running a container that needs to run inside the dind:
docker run --rm --name test --gpus=all --runtime=nvidia nvidia/cuda:12.0.1-base-ubuntu22.04 nvidia-smi
I get the following error:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy' nvidia-container-cli: mount error: stat failed: /proc/driver/nvidia/gpus/0000:04:00.0: no such file or directory: unknown.