harry-rogers / wgisd-faster-r-cnn Goto Github PK

I am using a 2080 super GPU and after the training loop I end up with this error

Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

Fair enough, If I follow the error and introduce .backward(retain_graph = True),

CUDA just demands for enormous memory like

CUDA out of memory. Tried to allocate 133.05 GiB (GPU 0; 7.79 GiB total capacity; 444.44 MiB already allocated; 5.24 GiB free; 612.00 MiB reserved in total by PyTorch)(Edited)

Could you share any possible fix ?