how to use GPU about docker-python HOT 9 CLOSED

@nerdcha I see that there is script to build GPU image. But Is there any documentation how could I pull GPU image without assembling it from the source code?

from docker-python.

Think, you might need to use nvidia-docker to make use of your gpu.

from docker-python.

GPU support is something that we're going to add soon.

from docker-python.

I was successfully able to build and run the GPU image from gpu.Dockerfile.

However, running a GPU script inside that image I hit this:

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=74 error=35 : CUDA driver version is insufficient for CUDA runtime version
Traceback (most recent call last):
  File "kesnet.py", line 115, in <module>
    display_imgs(np.asarray(md.trn_ds.denorm(x)))
  File "/opt/conda/lib/python3.6/site-packages/fastai/dataset.py", line 252, in denorm
    if type(arr) is not np.ndarray: arr = to_np(arr)
  File "/opt/conda/lib/python3.6/site-packages/fastai/core.py", line 60, in to_np
    if isinstance(v, torch.cuda.HalfTensor): v=v.float()
RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp:74

I am able to run this script on Kaggle.com and outside of the Docker image.

I've tried on 2 machines. One has Nvidia Driver Version: 410.48 and the other has Nvidia Driver Version: 396.54.

I'm not an expert at Docker or CUDA so could be something simple I've got configured wrong, but thought I'd post here in case there's an issue or the GPU Docker isn't ready for mass consumption yet.

from docker-python.

Hi @breckuh,

Both Nvidia driver version should be fine.

Have you updated any packages inside your GPU image?

from docker-python.

@rosbo , I have not.

I just downloaded one of the base Nvidia Docker images and was not able to detect the GPU inside even that container ("CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected"), so my issue appears to be more of a system wide one and not with this Kaggle image per se. I will keep digging. Thanks!

from docker-python.

@breckuh are you using the docker NVIDIA runtime?
https://github.com/NVIDIA/nvidia-docker

# you will need to add some apt repos first, see link above
sudo apt-get install -y nvidia-docker2

docker run --runtime=nvidia --rm kaggle/python-build-gpu /bin/bash

from docker-python.

@rosbo I am not! Thanks. Attempting to install now (but running into unrelated typical dependency problems :) ).

from docker-python.

I was able to get the docker NVIDIA runtime installed, able to run jobs on the GPU in the base NVIDIA CUDA devel image, and can now see the GPUs with nividia-smi from within the running kaggle/python-build-gpu image, but still get the "CUDA driver version is insufficient for CUDA runtime version" error message, same as above.

Still investigating....

from docker-python.