zielon / insta-pytorch Goto Github PK
View Code? Open in Web Editor NEWINSTA - Instant Volumetric Head Avatars [Demo]
Home Page: https://zielon.github.io/insta/
License: Other
INSTA - Instant Volumetric Head Avatars [Demo]
Home Page: https://zielon.github.io/insta/
License: Other
Thanks for your pytorch version! While after the training start, it says there NaN or Inf found in input tensor, and Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds"
failed. The output is listed below:
python main_insta.py data/wojtek_1 --workspace workspace/wojtek_1 -O --tcnn (insta-pytorch)
Namespace(path='data/wojtek_1', O=True, test=False, workspace='workspace/wojtek_1', seed=49, iters=100000, val_freq=10, lr=0.0025, ckpt='latest', num_rays=4096, cuda_ray=True, max_steps=1024, num_steps=1024, upsample_steps=0, update_extra_interval=16, max_ray_batch=4096, patch_size=1, fp16=True, ff=False, tcnn=True, color_space='srgb', preload=True, bound=2, scale=1, offset=[0, 0, 0], dt_gamma=0.00390625, min_near=0.01, density_thresh=10, bg_radius=-1, gui=False, W=512, H=512, radius=5, fovy=50, max_spp=64, error_map=False, clip_text='', rand_pose=-1)
=========== NETWORK TCNN ===========
NeRFNetwork(
(encoder): Encoding(n_input_dims=3, n_output_dims=128, seed=1337, dtype=torch.float16, hyperparams={'base_resolution': 16, 'hash': 'CoherentPrime', 'interpolation': 'Linear', 'log2_hashmap_size': 18, 'n_features_per_level': 8, 'n_levels': 16, 'otype': 'Grid', 'per_level_scale': 1.4472692012786865, 'type': 'Hash'})
(sigma_net): Network(n_input_dims=144, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 2, 'n_neurons': 64, 'otype': 'FullyFusedMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
(encoder_dir): Encoding(n_input_dims=3, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'degree': 4, 'otype': 'SphericalHarmonics'})
(color_net): Network(n_input_dims=31, n_output_dims=3, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 3, 'n_neurons': 64, 'otype': 'FullyFusedMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
)
data/wojtek_1
data/wojtek_1/canonical.obj
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1500/1500 [00:38<00:00, 39.42it/s]
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
/home/wangzy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/home/wangzy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
warnings.warn(msg)
Loading model from: /home/wangzy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/lpips/weights/v0.1/alex.pth
[INFO] Trainer: ngp | 2023-04-07_15-40-45 | cuda | fp16 | workspace/wojtek_1
[INFO] #parameters: 26590464
[INFO] Loading latest checkpoint ...
[WARN] No checkpoint found, model randomly initialized.
data/wojtek_1
data/wojtek_1/canonical.obj
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 43.93it/s]
[mark untrained grid] 4194084 from 4194304
==> Start Training Epoch 1, lr=0.002500 ...
0% 0/1500 [00:00<?, ?it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: : 0% 1/1500 [00:00<06:20, 3.94it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: : 0% 1/1500 [00:00<06:20, 3.94it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: : 0% 2/1500 [00:00<06:20, 3.94it/s]NaN or Inf found in input tensor.
loss=353813397595729098627955600845176832.0000 (nan), lr=0.002500: : 0% 5/1500 [00:00<01:33, 15.93it/s]NaN or Inf found in input tensor.
loss=25.2939 (nan), lr=0.002500: : 0% 6/1500 [00:00<01:33, 15.93it/s]NaN or Inf found in input tensor.
loss=0.1733 (nan), lr=0.002500: : 1% 10/1500 [00:00<01:07, 22.14it/s] NaN or Inf found in input tensor.
loss=459.3705 (nan), lr=0.002500: : 1% 13/1500 [00:00<00:58, 25.51it/s]NaN or Inf found in input tensor.
loss=nan (nan), lr=0.002500: : 1% 13/1500 [00:00<00:58, 25.51it/s] NaN or Inf found in input tensor.
loss=1.0937 (nan), lr=0.002500: : 1% 15/1500 [00:00<00:58, 25.51it/s]../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [52,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [120,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [46,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [47,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [48,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [49,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [50,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [51,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [52,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [53,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [54,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [55,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [56,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [58,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [59,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [60,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [61,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [62,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [8,0,0], thread: [63,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
terminate called after throwing an instance of 'thrust::system::system_error'
what(): CUDA free failed: cudaErrorAssert: device-side assert triggered
fish: Job 1, 'python main_insta.py data/wojte…' terminated by signal SIGABRT (Abort)
If there's any other information I need to provide, please let me know!
line 66 in bvh/setup.py : -DNUM_THREADS=256, it causes the error:
block_merge_sort.cuh(169): error: expected a "," or ">"
block_merge_sort.cuh use NUM_THREADS as an input param. And I commented out this line in setup.py, it works.
Ubuntu 22.04.3 LTS
python 3.10
torch 2.1.2
cuda 12.1
Hi Zielon,
When I used your codes to train INSTA on my own video, an error occurred. It is important to note that to reduce VRAM requirements I disabled -O
and just used --fp16 --cuda_ray
according to #2.
Below is the error report.
0% 0/1500 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/wanghy/INSTA-pytorch/main_insta.py", line 160, in <module>
trainer.train(train_loader, valid_loader, max_epoch)
File "/home/wanghy/INSTA-pytorch/insta/utils.py", line 625, in train
self.train_one_epoch(train_loader)
File "/home/wanghy/INSTA-pytorch/insta/utils.py", line 838, in train_one_epoch
self.model.update_extra_state(loader)
File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/wanghy/INSTA-pytorch/insta/renderer.py", line 572, in update_extra_state
cas_xyzs, faces = self.warp_position(triangles, cas_xyzs)
File "/home/wanghy/INSTA-pytorch/insta/renderer.py", line 293, in warp_position
distances, closest_points, closest_faces, closest_bcs = m(triangles, xyzs.unsqueeze(dim=0))
File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/wanghy/INSTA-pytorch/bvh/bvh_distance_queries/bvh_search_tree.py", line 109, in forward
output = BVHFunction.apply(
File "/home/wanghy/miniconda3/envs/insta-pytorch/lib/python3.9/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/wanghy/INSTA-pytorch/bvh/bvh_distance_queries/bvh_search_tree.py", line 42, in forward
outputs = bvh_distance_queries_cuda.distance_queries(
RuntimeError: triangles must be a CUDA tensor
PS: My machine is NVIDIA G-RTX 3090 with 24GB VRAM and over 200G RAM, so I am also confused why it cannot meet the memory requirement when using -O
.
I've been trying to train the models on the wojtek data, on FP32, since I want to gather data for a research project and fully fused MLPs obscure weight data. However, when I train on FP32, the resulting images after evaluation and testing are blurry and are not the same as the images produced by the TCNN, FP16 and fully fused model. Is this behaviour expected, or is something wrong with the code?
there is no network_ff file in ./insta folder, but there is option for ff in main_insta.py line 80
if opt.ff:
opt.fp16 = True
assert opt.bg_radius <= 0, "background model is not implemented for --ff"
from insta.network_ff import NeRFNetwork
Hi Zielon, thank you for your amazing work!
I wanted to ask if there is a missing module in the repository. When I run the INSTA-pytorch code, I receive an error message stating "No module named 'bvh_distance_queries_cuda'". I have searched both your repository and the BVH repository, but I cannot find any module with that name. Could you please advise me on how to resolve this issue?
Looking forward to your reply!
I trained a model with provided Wojtek data, inference speed is around 1.4 seconds per frame.
I also trained a model using different data which I generated following the steps mentioned. However, it takes 7 seconds to infer 1 frame. I used same amount of num_rays. What might be causing this slowness?
I have 32 GB of RAM and Tesla T4 with 16 GB VRAM.
I see that 24GB of VRAM is required. Disabling -O
and just using --fp16 --cuda_ray
seem to reduce VRAM requirements.
However, I am running out of regular memory during data preparation at the start of third epoch.
What is the memory requirements with this setting?
I was wondering the reasoning behind this change. As far as I know, batch_rodrigues method was being used in FLAME to compute rotation matrix. I also see that full_pose has 30 elements instead of 15.
I have a project which uses a FLAME version that uses batch_rodrigues, I am unable to use it due to this change. Is there a way to coınvert these changes?
Hello, thank you for the great work!
I encounter a cuda error when running the command line python main_insta.py data/test/ --workspace workspace/test/ -O --tcnn
Namespace(H=256, O=True, W=256, bg_radius=-1, bound=2, ckpt='latest', clip_text='', color_space='srgb', cuda_ray=True, density_thresh=10, dt_gamma=0.00390625, error_map=False, ff=False, fovy=50, fp16=True, gui=False, iters=100000, lr=0.0025, max_ray_batch=4096, max_spp=64, max_steps=1024, min_near=0.01, num_rays=4096, num_steps=1024, offset=[0, 0, 0], patch_size=16, path='data/test/', preload=True, radius=5, rand_pose=-1, scale=1, seed=49, tcnn=True, test=False, update_extra_interval=16, upsample_steps=0, val_freq=5, workspace='workspace/test/')
=========== NETWORK TCNN ===========
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
Warning: FullyFusedMLP is not supported for the selected architecture 70. Falling back to CutlassMLP. For maximum performance, raise the target GPU architecture to 75+.
NeRFNetwork(
(encoder): Encoding(n_input_dims=3, n_output_dims=128, seed=1337, dtype=torch.float16, hyperparams={'base_resolution': 16, 'hash': 'CoherentPrime', 'interpolation': 'Linear', 'log2_hashmap_size': 18, 'n_features_per_level': 8, 'n_levels': 16, 'otype': 'Grid', 'per_level_scale': 1.4472692012786865, 'type': 'Hash'})
(sigma_net): Network(n_input_dims=144, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 2, 'n_neurons': 64, 'otype': 'CutlassMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
(encoder_dir): Encoding(n_input_dims=3, n_output_dims=16, seed=1337, dtype=torch.float16, hyperparams={'degree': 4, 'otype': 'SphericalHarmonics'})
(color_net): Network(n_input_dims=31, n_output_dims=3, seed=1337, dtype=torch.float16, hyperparams={'encoding': {'offset': 0.0, 'otype': 'Identity', 'scale': 1.0}, 'network': {'activation': 'ReLU', 'n_hidden_layers': 3, 'n_neurons': 64, 'otype': 'CutlassMLP', 'output_activation': 'None'}, 'otype': 'NetworkWithInputEncoding'})
)
100%|████████████████████████████████████████████████████████████████████████████████████| 1463/1463 [01:06<00:00, 22.15it/s]
Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
Loading model from: /root/.local/conda/envs/insta/lib/python3.8/site-packages/lpips/weights/v0.1/alex.pth
Setting up [LPIPS] perceptual loss: trunk [vgg], v[0.1], spatial [off]
Loading model from: /root/.local/conda/envs/insta/lib/python3.8/site-packages/lpips/weights/v0.1/vgg.pth
[INFO] Trainer: ngp | 2023-04-23_18-25-34 | cuda | fp16 | trails/trail_woman_1_4/
[INFO] #parameters: 26590464
[INFO] Loading latest checkpoint ...
[WARN] No checkpoint found, model randomly initialized.
100%|████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 28.17it/s]
[mark untrained grid] 3998599 from 4194304
==> Start Training Epoch 1, lr=0.002500 ...
0% 0/1463 [00:00<?, ?it/s]/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [1,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [2,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [3,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [4,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [5,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [6,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [7,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [4,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [197,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [197,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [197,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [8,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [9,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [10,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [11,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [12,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [13,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [77,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [115,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [116,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [117,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [118,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [119,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [120,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [121,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [122,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [123,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [124,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [28,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [342,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [302,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [302,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize
failed.
/opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [302,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize
failed.
Traceback (most recent call last):
File "main_insta.py", line 160, in
trainer.train(train_loader, valid_loader, max_epoch)
File "/root/INSTA/insta/utils.py", line 626, in train
self.train_one_epoch(train_loader)
File "/root/INSTA/insta/utils.py", line 839, in train_one_epoch
self.model.update_extra_state(loader)
File "/root/.local/conda/envs/insta/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/root/INSTA/insta/renderer.py", line 572, in update_extra_state
cas_xyzs, faces = self.warp_position(triangles, cas_xyzs)
File "/root/INSTA/insta/renderer.py", line 295, in warp_position
canonical_triangles = torch.index_select(self.canonical_triangles, 1, closest_faces[0])
RuntimeError: CUDA error: device-side assert triggered
0% 0/1463 [00:01<?, ?it/s]
Hello,
I trained a NeRF using the original INSTA repository, and have my trained weights. How do I convert those weights to something I can use in INSTA-pytorch? (ie. how do I convert snapshot.msgpack to .pth?)
Thank you!
Hi,
Thanks for your excellent work., I'm interested in the amazing results. However, i found sth was missed in the code when i run the project. In insta/masks.py, '/home/wzielonka/CLionProjects/real-time-avatar/mesh_masks/lips.obj' was not included, can you fix it, thank you very much!
Traceback (most recent call last):
File "/INSTA-pytorch/main_insta.py", line 132, in
trainer.save_mesh(resolution=256, threshold=10)
File "/INSTA-pytorch/insta/utils.py", line 602, in save_mesh
vertices, triangles = extract_geometry(self.model.aabb_infer[:3], self.model.aabb_infer[3:], resolution=resolution, threshold=threshold, query_func=query_func)
File "/INSTA-pytorch/insta/utils.py", line 182, in extract_geometry
u = extract_fields(bound_min, bound_max, resolution, query_func)
File "/INSTA-pytorch/insta/utils.py", line 175, in extract_fields
val = query_func(pts).reshape(len(xs), len(ys), len(zs)).detach().cpu().numpy() # [S, 1] --> [x, y, z]
File "/INSTA-pytorch/insta/utils.py", line 599, in query_func
sigma = self.model.density(pts.to(self.device))['sigma']
TypeError: density() missing 1 required positional argument: 'cond'
is there a way to export the mesh (.obj) and mesh texture (wrapped) for the avatar
want to export output for use in 3d application so .obj, .fbx + mesh needed
Thanks for the nice code!
I have a question about the definition of depth.
raymarching.cu
, it seems that the rendered depth (calculated from deltas[1]) is the distance along the rays_d (point distance).
utils.py
, L1Loss is directly applied to outputs['depth'] and data['depth'], does it mean that the data['depth'] is actually point distance instead of the usually used plane distance?I am trying to generate the mesh after training. There is a cond parameter in the model.density function that is not passed in line 599 in utils.py.
sigma = self.model.density(pts.to(self.device), cond=1)['sigma']
INSTA-pytorch/freqencoder/src/freqencoder.cu
INSTA-pytorch/gridencoder/src/gridencoder.cu
could you provide torch version source code for above files?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.