cong-yi / dmm Goto Github PK

surf (1000000, 5) 
 [[ 1.546e-01  1.385e-01  9.780e-02 -1.158e-04  0.000e+00]
 [ 3.004e-01  2.991e-02 -2.903e-02 -1.121e-04  0.000e+00]
 [-3.631e-01 -2.900e-02  3.725e-01  1.944e-04  0.000e+00]
 ...
 [-3.717e-01 -6.777e-02 -7.662e-03 -2.631e-04  0.000e+00]
 [-2.598e-01  3.294e-02  5.223e-02 -1.512e-04  0.000e+00]
 [ 2.061e-01  3.635e-02  4.215e-01 -2.431e-04  0.000e+00]]

pos (583544, 5) 
 [[-0.765 -0.643 -0.876  0.829 -1.   ]
 [ 0.395 -0.938 -0.854  1.051 -1.   ]
 [ 0.826 -0.398  0.209  0.489 -1.   ]
 ...
 [ 0.431 -0.804 -0.435  0.771 -1.   ]
 [-0.758  0.446 -0.968  0.713 -1.   ]
 [ 0.831 -0.763  0.232  0.784 -1.   ]]
neg (116456, 5) 
 [[-0.087  0.899  0.095 -0.714 -1.   ]
 [-0.333  0.431 -0.352 -0.361 -1.   ]
 [ 0.441  0.299  0.066 -0.125 -1.   ]
 ...
 [-0.55   0.818  0.803 -0.728 -1.   ]
 [ 0.041  0.884 -0.198 -0.724 -1.   ]
 [-0.375  0.226 -0.034 -0.211 -1.   ]]
normal (1000000, 3) 
 [[-0.695 -0.671 -0.26 ]
 [-0.813 -0.581  0.04 ]
 [-0.853 -0.318  0.414]
 ...
 [ 0.573 -0.231  0.787]
 [ 0.745 -0.657 -0.115]
 [-0.454 -0.784 -0.424]]

Traceback (most recent call last):
  File "/home/xxl/DMM/train_dmm.py", line 410, in <module>
    main_function(args.experiment_directory, args.continue_from)
  File "/home/xxl/DMM/train_dmm.py", line 336, in main_function
    train_loss, losses_log = dmm_net(latent_vecs, sdf_data, is_on_surf, normal, centers_tensor, indices)
  File "/home/xxl/anaconda3/envs/dmm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxl/DMM/networks/dmm_net.py", line 151, in forward
    losses = component_sdf_normal_loss(component_sdf, coords, is_on_surf, normal,
  File "/home/xxl/DMM/networks/loss.py", line 9, in component_sdf_normal_loss
    gradient = compute_gradient(pred_sdf, xyz)
  File "/home/xxl/DMM/utils/math.py", line 16, in compute_gradient
    grad = torch.autograd.grad(y, [x], grad_outputs=grad_outputs, create_graph=True)[0]
  File "/home/xxl/anaconda3/envs/dmm/lib/python3.9/site-packages/torch/autograd/__init__.py", line 276, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 23.70 GiB total capacity; 22.67 GiB already allocated; 6.88 MiB free; 22.70 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF