apple / ml-pgdvs Goto Github PK

Hello, when I try to run "Spatial Temporal Interpolation Visualizations" on the benchmark of nvidia dynamic scenes, I got a problem like this:

gnt: 91%|█████████ | 70/77 [2:55:22<17:32, 150.33s/it] gnt: 92%|█████████▏| 71/77 [2:57:54<15:04, 150.74s/it] gnt: 94%|█████████▎| 72/77 [3:00:22<12:30, 150.04s/it] gnt: 95%|█████████▍| 73/77 [3:02:54<10:02, 150.56s/it] gnt: 96%|█████████▌| 74/77 [3:05:23<07:30, 150.10s/it] gnt: 97%|█████████▋| 75/77 [3:07:51<04:58, 149.32s/it] gnt: 99%|█████████▊| 76/77 [3:10:19<02:29, 149.11s/it] gnt: 100%|██████████| 77/77 [3:12:23<00:00, 149.92s/it] Error executing job with overrides: ['verbose=true', 'distributed=false', 'seed=0', 'resume=vis_wo_resume', 'resume_dir=null', 'engine=visualizer_pgdvs', 'model=pgdvs_renderer', 'model.softsplat_metric_abs_alpha=100.0', 'static_renderer=gnt', 'static_renderer.model_cfg.ckpt_path=/data/code/ml-pgdvs/ckpts/gnt/model_720000.pth', 'series_eval=false', 'eval_batch_size=1', 'n_max_eval_data=-1', 'eval_save_individual=true', 'engine.engine_cfg.render_cfg.render_stride=1', 'engine.engine_cfg.render_cfg.chunk_size=2048', 'engine.engine_cfg.render_cfg.sample_inv_uniform=true', 'engine.engine_cfg.render_cfg.n_coarse_samples_per_ray=256', 'engine.engine_cfg.render_cfg.n_fine_samples_per_ray=0', 'engine.engine_cfg.render_cfg.mask_oob_n_proj_thres=1', 'engine.engine_cfg.render_cfg.mask_invalid_n_proj_thres=4', 'engine.engine_cfg.render_cfg.dyn_pcl_remove_outlier=true', 'engine.engine_cfg.render_cfg.dyn_pcl_outlier_knn=50', 'engine.engine_cfg.render_cfg.dyn_pcl_outlier_std_thres=0.1', 'engine.engine_cfg.render_cfg.gnt_use_dyn_mask=true', 'engine.engine_cfg.render_cfg.gnt_use_masked_spatial_src=false', 'engine.engine_cfg.render_cfg.dyn_render_use_flow_consistency=false', 'dataset=combined', 'dataset.dataset_list.train=[nvidia_eval]', 'dataset.dataset_list.eval=[nvidia_eval]', 'dataset.dataset_list.vis=[nvidia_vis]', 'dataset.dataset_specifics.mono_vis.scene_ids=[Balloon1]', 'dataset.data_root=/data/code/ml-pgdvs/data', 'n_dataloader_workers=1', 'dataset_max_hw=-1', 'dataset.use_aug=false', 'dataset.dataset_list.vis=[nvidia_vis]', 'dataset.dataset_specifics.nvidia_vis.scene_ids=[Balloon1]', 'vis_specifics.n_render_frames=400', 'vis_specifics.vis_center_time=50', 'vis_specifics.vis_time_interval=50', 'vis_specifics.vis_bt_max_disp=32'] Traceback (most recent call last): File "/data/code/ml-pgdvs/pgdvs/run.py", line 267, in <module> cli() File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main _run_hydra( File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra _run_app( File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/_internal/utils.py", line 452, in _run_app run_and_report( File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/_internal/utils.py", line 216, in run_and_report raise ex File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/_internal/utils.py", line 213, in run_and_report return func() File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/_internal/utils.py", line 453, in <lambda> lambda: hydra.run( File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run _ = ret.return_value File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/data/code/ml-pgdvs/pgdvs/run.py", line 263, in cli run(cfg, hydra_config) File "/data/code/ml-pgdvs/pgdvs/run.py", line 192, in run return _distributed_worker( File "/data/code/ml-pgdvs/pgdvs/run.py", line 146, in _distributed_worker output = engine.run() File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/data/code/ml-pgdvs/pgdvs/engines/visualizer_pgdvs.py", line 26, in run self.vis_model() File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/data/code/ml-pgdvs/pgdvs/engines/visualizer_pgdvs.py", line 91, in vis_model ret_dict = self._get_model_module(self.model).forward( File "/data/code/ml-pgdvs/pgdvs/renderers/pgdvs_renderer.py", line 146, in forward (render_dyn_rgb, render_dyn_mask, render_dyn_info) = self.dyn_renderer( File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/data/code/ml-pgdvs/pgdvs/renderers/pgdvs_renderer_dyn.py", line 184, in forward splat_dyn_img_full, softsplat_metric_src1_to_src2 = self.softsplat_img( File "/data/code/ml-pgdvs/pgdvs/renderers/pgdvs_renderer_base.py", line 80, in softsplat_img splat_img_src1_to_tgt = softsplat.softsplat( File "/data/code/ml-pgdvs/pgdvs/utils/softsplat.py", line 311, in softsplat tenOut = softsplat_func.apply(tenIn, tenFlow) File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(*args, **kwargs) # type: ignore[misc] File "/data/apps/anaconda3/envs/pgdvs/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 121, in decorate_fwd return fwd(*args, **kwargs) File "/data/code/ml-pgdvs/pgdvs/utils/softsplat.py", line 421, in forward assert False AssertionError

I have followed the instruction of "https://github.com/apple/ml-pgdvs/blob/main/docs/BENCHMARK_NVIDIA.md" , and I just want to show the visuable result. I guess this may be related to the config process, but how could I solve this problem? Thank you!

Environment: CUDA 11.8 + nvidia_A800 and without jax

Hello, I notice that there is a pretrained gnt model in your download_ckpts.sh like this:

if [ "${FLAG_ORIGINAL}" == "1" ]; then
# GNT
if [ ! -f ${DATA_ROOT}/gnt/generalized_model_720000.pth ]; then
gdown 1AMN0diPeHvf2fw53IO5EE2Qp4os5SkoX -O ${DATA_ROOT}/gnt/
fi

However, I couldn't find this file on google drive and they tell me it not exist. How could I get this?
Thanks!