I used tensorflow-gpu==1.14.0 , tensorflow-tensorboard==0.1.7 cuda=10, cudnn=7.6.3, GPU is GeForce RTX 2070 SUPER memory is 8GB, and the hardware of my Nvidia driver is 430.64 .When I python train.py, I used the batch size 1, but the memory is already to 7.370GB, and then get this error?
I think it's out of memory?
What else can I modify to train.py for less memory?
=========================================================================
2020-04-21 13:53:25.887229: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-04-21 13:53:25.914730: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-04-21 13:53:25.916246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2070 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:01:00.0
2020-04-21 13:53:25.944964: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-04-21 13:53:26.535817: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-04-21 13:53:26.944722: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-04-21 13:53:26.990557: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-04-21 13:53:27.168829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-04-21 13:53:27.320085: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-04-21 13:53:27.695659: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Traceback (most recent call last):
File "/home/muyun/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/muyun/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/muyun/anaconda3/envs/test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node segnet/encoder/block_1/conv_1/Conv2D}}]]
[[acc/Mean/_273]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node segnet/encoder/block_1/conv_1/Conv2D}}]]
0 successful operations.
0 derived errors ignored.