Getting training loss value as nan and val loss as nan about neural-audio-fp HOT 8 CLOSED

@kasireddygariDineshKumarReddy
It seems FAISS requires to be recompiled with the 'sm_86' tag. At the time I was compiling it, the gpu type was not taken into account. I recommend you to try conda install faiss-gpu or faiss-cpu(search speed is slow) first. If conda install faiss-gpu doesn't work, you will have to build from source as described in: https://github.com/facebookresearch/faiss/blob/main/INSTALL.md.

from neural-audio-fp.

I had ran conda install faiss-gpu successfully but cant get rid of nan loss

from neural-audio-fp.

@kasireddygariDineshKumarReddy
Sorry, forget my answer. It's not a faiss related issue. The same issue as yours
was previously solved in #20.

from neural-audio-fp.

I have ran $docker pull mimbres/neural-audio-fp:cuda11.2.0-cudnn8 this command still problem is same. I'm still getting W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name' and Loss as NAN

from neural-audio-fp.

Your issue is related to specific GPU type, CUDA toolkit version and Driver version. So I cannot help you much for the basic setup.
https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html https://docs.nvidia.com/deploy/cuda-compatibility/index.html
According to the compatiability table, A5000 is Ampere architecture GPU (sm_80, sm_86, sm_87). Minimum Driver version is '450.36.06', and CUDA vesrion > 11.0. 11.0 is buggy, so 11.2 would be a good choice. 11.2 is also the most preferred version for tensorflow > 2.5.
I also recommend you to check first if any other tensorflow code runs correctly on your environment. If it works, you can manually install the requirements (of this repo) step by step.

from neural-audio-fp.

tensorflow/tensorflow#45044
reinstall tensorflow 2.6 may help...

from neural-audio-fp.

please let me clarify what are the updates to do in yaml file to avoid incompatibilities.

from neural-audio-fp.

Your frustration is understandable. However, I can't help you this way. First of all, can you check the results of the methods I recommended earlier? Are the other TensorFlow examples working well, the correct driver version is installed, etc. If that's clear, the training part of the repo can be run in any environment without this docker. Keep in mind that I'm limited in helping out with issues that I can't reproduce.

P.S. I also recommend you to try an official Tensorflow Docker image, and follow the instruction in this repo for manual installation.

from neural-audio-fp.