i have restarted pc and ubuntu multiple times.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.60.01              Driver Version: 551.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:01:00.0  On |                  N/A |
| 30%   54C    P0            121W /  350W |    1642MiB /  24576MiB |      4%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

from tensorflow.

@jeanswiegers If TF is not recognizing GPU, then could you please verify the build compatibility by running the following in your WSL environment;

import tensorflow as tf
print(tf.test.is_built_with_cuda())

This should output True. If it's False, you might need to reinstall TensorFlow with GPU support. For more information on WSL with GPU support please refer to https://www.tensorflow.org/install/pip. The TensorFlow version needs to be compatible with your CUDA version.

Thank you!

from tensorflow.

@jeanswiegers If TF is not recognizing GPU, then could you please verify the build compatibility by running the following in your WSL environment;

import tensorflow as tf
print(tf.test.is_built_with_cuda())

This should output True. If it's False, you might need to reinstall TensorFlow with GPU support. For more information on WSL with GPU support please refer to https://www.tensorflow.org/install/pip. The TensorFlow version needs to be compatible with your CUDA version.

Thank you!

Hi Sushreebarsa, thanks for your help. It does return True in my environment.

from tensorflow.

@jeanswiegers Thank you for your quick response!
Simply restarting your WSL instance (wsl --shutdown) or your entire computer can resolve environment variable issues. Could you please try this once. If the issue continues then, please use nvidia-smi within your WSL terminal to confirm your RTX3090 is recognized by the NVIDIA drivers. If not, there might be an issue with the driver installation itself.
Thank you!

from tensorflow.

I managed to get it working by installing the latest supported CUDA version (12.3) Ubuntu runfile stated on tensorflows website.

Only running
pip install tensorflow[with-cuda] doesn't work.

from tensorflow.

Working

from tensorflow.

Are you satisfied with the resolution of your issue?
Yes
No

from tensorflow.

@jeanswiegers Glad it worked fine for you.
Thank you!

from tensorflow.

Almost final and automated fix below

• Where I found the resolution

  • TF 2.16.1 Fails to work with GPUs
    • Solution proposed by "sh-shahrokhi", improved by "ChristofKaufmann"
    • See specially Comment by COntributor
  • Related Issues
    • GPU not detected on WSL2, where I have post some comments
    • Tensorflow WSL GPU CUDA recognition issue RTX3090
    • Once gain: tf.2.16.1 fails to recognize GPUs
  • Other mention on social media
    • Reddit issue

• Exact solution

  • Temporary fix (after activating environment in which Tensorflow 2.16.1 is installed)

      export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
      export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
      

  • Automating the variable set/unset process with Anaconda (one-time setup)
    • Activate your environement in which TF 2.16.1 is installed
    • Two files to be created in "anaconda3/envs/<ENV_NAME>/etc/conda"
      • anaconda3/envs/<ENV_NAME>/etc/conda/activate.d/env_vars.sh

        #!/bin/sh
        export NVIDIA_DIR=$(dirname $(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)")))
        export LD_LIBRARY_PATH=$(echo ${NVIDIA_DIR}/*/lib/ | sed -r 's/\s+/:/g')${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
        

      • anaconda3/envs/<ENV_NAME>/etc/conda/deactivate.d/env_vars.sh

        #!/bin/sh
        unset NVIDIA_DIR
        unset LD_LIBRARY_PATH
        

    • Official documentation to do this via conda.io
    • Stack-overflow question where I got this Set environment vars when activating conda env

• What else helped me

  • How to list all variables names and their current values?
  • Conda documenation for "env_vars.sh" for activate/deactivate

from tensorflow.