RunTimeError: CuDNN error: CUDNN_STATUS_SUCCESS when train the model about deepehr HOT 5 CLOSED

nyumedml commented on June 1, 2024

RunTimeError: CuDNN error: CUDNN_STATUS_SUCCESS when train the model

from deepehr.

Comments (5)

jingsliu commented on June 1, 2024

Can you attach the error log please? We used Linux machines so I'm not sure if the os is the issue

…

On May 3, 2019, at 12:27 PM, whungt ***@***.***> wrote: I'm trying to train the model with trainModel.py, following the command in ReadMe. But I've encountered the error in the title. I'm on win7 machine with anaconda. GPU: RTX 2070. My virtual env configuration: python 3.6.8 cudatoolkit 9.0 cudnn 7.3.1 pytorch 0.4.1 torchvision 0.2.1 This error occurs at the very beginning of the running. There's no concern of run out of memory or related problems. Someone help me on this? What's your configuration to run this successfully? Thanks in advance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

from deepehr.

whungt commented on June 1, 2024

@jingsliu

from deepehr.

jingsliu commented on June 1, 2024

Yeah looks like a purely CuDNN issue. I need to double check later but I believe we used pytorch 0.4.1 and CuDNN >= 8.

…

On May 3, 2019, at 5:54 PM, Jingshu Liu ***@***.***> wrote: Can you attach the error log please? We used Linux machines so I'm not sure if the os is the issue > On May 3, 2019, at 12:27 PM, whungt ***@***.***> wrote: > > I'm trying to train the model with trainModel.py, following the command in ReadMe. But I've encountered the error in the title. > > I'm on win7 machine with anaconda. GPU: RTX 2070. > My virtual env configuration: > python 3.6.8 > cudatoolkit 9.0 > cudnn 7.3.1 > pytorch 0.4.1 > torchvision 0.2.1 > > This error occurs at the very beginning of the running. There's no concern of run out of memory or related problems. > > Someone help me on this? What's your configuration to run this successfully? > Thanks in advance. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or mute the thread.

from deepehr.

whungt commented on June 1, 2024

Well...It seems that there is no cuDNN 8 yet. The latest one is cuDNN 7.5 for CUDA 10.1

from deepehr.

whungt commented on June 1, 2024

OK. It turns out that there seems to be some incompatible bugs between cuda/torch and some RTX GPUs.
I got the problem fixed by installing cuda92 in my virtual env:
conda install pytorch cuda92 -c pytorch
So now my configurations are:
win7 with RTX 2070, cuda 10.1.105
anaconda virtual env:
python 3.7.3
cudatoolkit 10.0.130
cudnn 7.3.1
cuda92 1.0
pytorch 1.1.0

Also, when I train the model, I need to add an exception line in enc_model.py:
Change line 773 to be:

try:
        train_loss += loss.data[0]
except IndexError:
        train_loss+=loss.data.item()

Similarly, change line 822 to be:

try:
        test_loss += (self.criterion(output, Disease)).data[0]
except IndexError:
        test_loss += (self.criterion(output, Disease)).data.item()

from deepehr.

RunTimeError: CuDNN error: CUDNN_STATUS_SUCCESS when train the model about deepehr HOT 5 CLOSED

Comments (5)

Related Issues (3)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent