ivrl / volrecon Goto Github PK

Hi,

Thanks for your excellent work.

I'm curious about reconstructing objects with single input view, like pixelNeRF. I know it's a highly ill-posed problem but I still wonder the performance in this setting (I noticed the ablation about source views in the appendix that show the result of 2 views at least).

Or any suggestions on how to improve the performance, such as fine tuning under single view setting?

Hi, thank you for your great work! On dtu datasets, it looks good. I try to extract the mesh on custom datasets. First, I use script/eval_general.sh to rendering image and depth. Then I try to run tsdf_fusion.sh to get the reconstructed meshes. Finally, I changed the image shape and view list to my own datasets in evaluation/clean_mesh.py. But results are badly. Here are some of my results, hope you can help me, thank you!

Hi, thanks for the great work. Could you please tell me what is the expected training time for the method and what GPU you used for the same?

Thanks in advance

Hello, thank you for sharing the code of your great work.
I was wondering if it's possible to fine-tune the pretrained model on a new scene rather than running inference, similar to how it is done in SparseNeuS. In the paper, there was no mention of this so I just assumed that it is not possible but I might be wrong.

Also, I was trying to run the model on a scene from BlendedMVS, but I could not get any meaningful reconstruction as seen here:

when looking at the rendered depth maps, I get something, but the mesh generation does not give good result:

I tried tuning the value of self.offset_dist but still no success. Do you have an idea of what could be wrong here ?

Thanks

Hi, thanks for the excellent work! Here I have a small question regarding the term 'source views' used in this project.

According to the code, the image to be rendered (denoted as 'gt-color' for brevity) is also included in 'source views', which means that the model aims to aggregate 'gt-color' itself along with some neighbor views to render the 'gt-color'. If this is the case, I wonder how can the efficacy of rgb-render-loss be gauranteed? Since the view transformer simply needs to pick the gt-color and gives zero weights to all other views. However, as suggested in Tab. 4 of the original paper, pure rgb-render-loss still delivers a reasonable result with a chamfer of 2.04.

Could you please clarify more on this point? Thanks in advance!

Thanks for publishing this great work! But we just found that there are something mismatches between our reproduction results and your paper. We adopt your released checkpoint to generate the mesh and test the chamfer distance of two image sets. We can only get the 1.48 average scores of these two sets (1.38 for the first set and 1.58 for the second set), and the results depicted in your paper is 1.38. So is this result reasonable? and the results in your paper is the result of the first set? or are we misunderstanding something?

Thanks for the great work! But I notice that as your paper said, you adopt the way of volume rendering SDF (Neus) to volume render SRDF. So how does this guarantee that the network learns the srdf of the sampling point instead of the sdf?

Hi! First of all, thanks for sharing this great work!
I assume that you are busy, but I have trivial questions about the camera matrix in dtu_train.py.

When you define w2c matrix in Line 270 of dtu_train.py, why do we need to multiply w2c_ref_inv matrix to all extrinsics?
w2cs.append(self.all_extrinsics[index_mat] @ w2c_ref_inv)
instead of simply using self.all_extrinsics[index_mat]?

I think this is because we need to make sample['ray_o'] defined in world coordinates like below.
sample['ref_pose'] = (intrinsics_pad @ sample['w2cs'])[0]
sample['ray_o'] = sample['ref_pose_inv'][:3,-1]

Am I correct here?

In addition, what is self.homo_pixel for? I assume that it is used to make ray_d in camera coordinate, but I am confused here.

Again, thanks for sharing the work and answering questions.

Thank you for your outstanding job. I have a question regarding the depth fusion and depth evaluation source code. Will you be releasing this code in the future?

I noticed that there is a depth_fusion.sh file, but I could not find the depth_fusion.py file. Could you please let me know where I can find the related source code? Additionally, it would be wonderful if you could provide a link to the depth evaluation source code as well.

Thank you in advance.

Thanks for sharing the great work!
When I was tried to reproduce your work, I found there is some blurring in the edge of the prediction image like following:

This image is generated by using the checkpoint you provided in the github.
And when I used the checkpoint trained with the DTU dataset by the bash script script/train_dtu.sh I still got the blurring images predicitons.

And I observed some inconsistency in the mesh (inconsistency betwent the checkpoint you provided in the github and the checkpoint after trained locally).

The left mesh is generated by the checkpoint you provided in the github, and the right one is generated by the model I trained locally.

Right now, I am confused with if there is some problems with my environment?
Or there will be some difference when running the code for several times?

Hi, thanks for your great work!
When I read the paper, I find it seems that there is a mistake in Appendix 7.

By the way, I wonder if there exists an algorithm to construct SRDF from depth images.
Thanks a lot~

My environment is Ubuntu 20.04, Python 3.8. It gives the error below when it tries to install pyembree:

Collecting pyembree (from -r requirements.txt (line 25))
Using cached pyembree-0.2.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
Using cached pyembree-0.2.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
Using cached pyembree-0.2.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
Using cached pyembree-0.2.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
Using cached pyembree-0.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
Using cached pyembree-0.2.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
Using cached pyembree-0.1.12-py3-none-any.whl (12.9 MB)
INFO: pip is still looking at multiple versions of pyembree to determine which version is compatible with other requirements. This could take a while.
Using cached pyembree-0.1.10.tar.gz (12.8 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [15 lines of output]
Traceback (most recent call last):
File "/home/voxar/anaconda3/envs/volrecon/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/home/voxar/anaconda3/envs/volrecon/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/voxar/anaconda3/envs/volrecon/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-nt016rlp/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-nt016rlp/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-nt016rlp/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "", line 33, in
ModuleNotFoundError: No module named 'build'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

how calculate camera parameter ?

Dear author,

After training on a large dataset, given a couple of images around an object, how soon can I use the pretrained weights to obtain the 3d mesh ?

Thanks!

Hello, thank you very much for your excellent work, I recently reproduced your results, but there were a series of problems during the data test, I wonder if you can provide some solutions.

When I tested with the test set you provided, none of the metrics here were shown, and I would like to know what is causing this. Also, I'd like to know if the quantitative analysis results in your article were taken from the visual interface or from the code

Hi, thanks for your great work!
I understand why true_cos in your implementaion is $-1$ because of SRDF.
ButI notice that, in your implementation, iter_cos in $renderer.py$ equals $1.5\cdot$true_cos when cos_anneal_ratio is equal to $1$, but $\cdot 1.5$ exceeds the upper bound of the cos function, which is different from implementation between NeuS and SparseNeus. I wonder if there are some reasons why you designed this.
Looking forward to your reply!

Note:
In VolRecon:

iter_cos = -(-true_cos * 0.5 + 0.5 * (1.0 - cos_anneal_ratio) - true_cos * cos_anneal_ratio)

In SparseNeus & NeuS:

iter_cos = -(F.relu(-true_cos * 0.5 + 0.5) * (1.0 - cos_anneal_ratio) + F.relu(-true_cos) * cos_anneal_ratio)

Hi, thanks for your amazing work. In the results section of the paper, a comparison with other methods such as sparseneus and mvsnerf has been done. I wanted to enquire about the results for mvsnerf and how you acquired them. I tried running marching cubes mvsnerf with a density threshold of 10 and am not getting satisfactory results. If possible, I'd be also grateful if you would be willing to share the code to do the same.

Thank you in advance

I ran the 'colmap_input.py' command post sparse reconstruction on custom data, resulting in camera files and pair.txt. Subsequently, I executed the script/eval_general.sh command, which created three folders, each containing data. However, when I try to execute 'tsdf_fusion.sh', I encountered 'Found Scans: []'. Can you please advise me on the necessary steps or if I missed something?

Hello, thank you very much for your excellent work, but when I was about to reproduce your work recently, I got an error that showed "loss=nan", I was training on the A5000GPU, what is the reason for this?

Hi, thanks for the great work，I had some problems replicating the paper, when i run eval_dtu.sh, main.py has an error . on line 183, trainer.validate(volrecon, dataloader_test1).
[rank0]: Traceback (most recent call last):
[rank0]: File "/data/tx/volrecon/VolRecon-main/main.py", line 183, in
[rank0]: trainer.validate(volrecon, dataloader_test1)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 697, in validate
[rank0]: return call._call_and_handle_interrupt(
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
[rank0]: return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
[rank0]: return function(*args, **kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 745, in _validate_impl
[rank0]: results = self._run(model, ckpt_path=self.ckpt_path)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
[rank0]: results = self._run_stage()
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1188, in _run_stage
[rank0]: return self._run_evaluate()
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1228, in _run_evaluate
[rank0]: eval_loop_results = self._evaluation_loop.run()
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
[rank0]: self.advance(*args, **kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
[rank0]: dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
[rank0]: self.advance(*args, **kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
[rank0]: output = self._evaluation_step(**kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
[rank0]: output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1494, in _call_strategy_hook
[rank0]: output = fn(*args, **kwargs)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 363, in validation_step
[rank0]: return self.model.validation_step(*args, **kwargs)
[rank0]: File "/data/tx/volrecon/VolRecon-main/code/model.py", line 265, in validation_step
[rank0]: self.extract_geometry(batch, batch_idx)
[rank0]: File "/data/tx/volrecon/VolRecon-main/code/model.py", line 375, in extract_geometry
[rank0]: srdf, points_x, depth, rgb = self.infer(batch=batch, ray_idx=ray_idx, source_imgs_feat=source_imgs_feat,
[rank0]: File "/data/tx/volrecon/VolRecon-main/code/model.py", line 143, in infer
[rank0]: rgb, depth, srdf, opacity, weight, points_in_pixel, _ = self.sample2rgb(batch, points_x, z_val, ray_d, ray_idx,
[rank0]: File "/data/tx/volrecon/VolRecon-main/code/model.py", line 85, in sample2rgb
[rank0]: rgb, depth, opacity, weight, variance = self.renderer.render(rearrange(z_val, "B RN SN -> (B RN) SN"),
[rank0]: File "/data/tx/volrecon/VolRecon-main/code/utils/renderer.py", line 32, in render
[rank0]: iter_cos = -(F.relu(-true_cos * 0.5 + 0.5) * (1.0 - cos_anneal_ratio) + F.relu(-true_cos) * cos_anneal_ratio)
[rank0]: File "/data/tx/Anaconda3/envs/UFORecon/lib/python3.10/site-packages/torch/nn/functional.py", line 1500, in relu
[rank0]: result = torch.relu(input)
[rank0]: TypeError: relu(): argument 'input' (position 1) must be Tensor, not float
I can't solve the problem right now, and I'd appreciate your help