It shows cuda_error_out_of_memory when I trained the network on nvidia1080ti. about dped HOT 4 CLOSED

Hi @KrisChou

Does it crash after this error?
If not, just skip this error, otherwise try reducing the batch size (from 50 to e.g. 30).

from dped.

batch size set to 30 doesnt work.By the way，i try to reduce the train size，all the generate loss is nan when I train the network.

from dped.

The above error appears when you GPU doesn't have enough RAM to run the network, and the only way to solve it is to reduce the batch size. Additionally, it looks like in your case GPU's RAM is also consumed by another process (only 9 out of 11GB are available), this can also lead to this error.

In general, 11-12GB should be definitely enough to run the network.

from dped.

Thanks

from dped.