angeladai / sgnn Goto Github PK

[CVPR'20] SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans

License: Other

Python 34.29% C++ 50.37% HLSL 0.24% C 15.10%

3d-reconstruction computer-vision computer-graphics deep-learning self-supervised-learning

sgnn's Introduction

SG-NN

SG-NN presents a self-supervised approach that converts partial and noisy RGB-D scans into high-quality 3D scene reconstructions by inferring unobserved scene geometry. For more details please see our paper SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans.

Code

Installation:

Training is implemented with PyTorch. This code was developed under PyTorch 1.1.0, Python 2.7, and uses SparseConvNet.

For visualization, please install the marching cubes by python setup.py install in marching_cubes.

Training:

See python train.py --help for all train options.
Example command: python train.py --gpu 0 --data_path ./data/completion_blocks --train_file_list ../filelists/train_list.txt --val_file_list ../filelists/val_list.txt --save_epoch 1 --save logs/mp --max_epoch 4
Trained model: sgnn.pth (7.5M)

Testing

See python test_scene.py --help for all test options.
Example command: python test_scene.py --gpu 0 --input_data_path ./data/mp_sdf_vox_2cm_input --target_data_path ./data/mp_sdf_vox_2cm_target --test_file_list ../filelists/mp-rooms_val-scenes.txt --model_path sgnn.pth --output ./output --max_to_vis 20

Data:

Scene data:
- mp_sdf_vox_2cm_input.zip (44G)
- mp_sdf_vox_2cm_target.zip (58G)
Train data:
- completion_blocks.zip (88G)
GenerateScans depends on the mLib library.

Citation:

If you find our work useful in your research, please consider citing:

@inproceedings{dai2020sgnn,
 title={SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans},
 author = {Dai, Angela and Diller, Christian and Nie{\ss}ner, Matthias},
 booktitle = {Proc. Computer Vision and Pattern Recognition (CVPR), IEEE},
 year = {2020}
}

sgnn's People

Contributors

Stargazers

Watchers

sgnn's Issues

Inference Requirements - Time and Memory

Hi @angeladai thank you for sharing :)

From your paper, I have two questions:

Do you have a rough estimate on inference time? I.e how long does one forward pass of a 64 x 64 x 128 volume at 2cm resolution take to process on your hardware.
Do you also have an estimate of how much memory is consumed while doing so?

Best regards and great work 👍 🎉
Magnus

The outputs are not correct in testing

Hi, I use the provided pretrained model in testing, but the output sdf looks strange. I print the output_sdf as follows:
[ 0 | 20 ] ['Z6MFQCViBuw_room7__0__'] (128, 416, 384) input_dim [128 416 384]
output_sdf [tensor([[ 0, 0, 0, 0],
[ 0, 0, 1, 0],
[ 0, 1, 0, 0],
...,
[ 47, 374, 327, 0],
[ 47, 375, 326, 0],
[ 47, 375, 327, 0]]), tensor([[-0.0468],
[-0.0468],
[-0.0468],
...,
[-0.0468],
[-0.0468],
[-0.0468]], device='cuda:0')]
It seems all the values in the output are the same -0.0468, so the output ply file is empty.
The only changes I made to the code is in the model.py line 365
I change
self.encoder.process_sparse[0].p0.spatial_size[k] = input_max_dim[k]
to
self.encoder.process_sparse[0].p0.spatial_size[k] = torch.tensor(input_max_dim[k])
As there is a type error when I use pytorch 1.3.
I use pytorch 1.3, cuda 10.2, python 3.6, SparseConvNet 2.0.
Could you give me any suggestions? Thank you so much.

Out of index error

Hi, there is an out of index error when I run the training code.
Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
File "/sgnn/torch/loss.py", line 67, in compute_bce_sparse_dense
tgtvalues = dense_tgts.view(-1)[flatlocs]
The log shows dense_tgts.view(-1) is of size 8192 while [flatlocs] is of size 32768
Could you give any suggetsions? I use the default settings during the training with pytorch 1.4 and python 3.6.

How to create my own sdf file for testing purposes ?

Hello,
First of all, I would like to say thanks for uploading the code and data. However, I would like to test model performance on the real scan using my mobile phone.

My plan is to use a smartphone to capture both RGB and depth images and then convert them to a projective truncated signed distance function (TSDF) volume. At the moment, I am using this tool to do it. However, the output of this tool is a mesh in ply format.

Therefore, I would like to ask how can I convert any mesh to the .sdf files which you are using in your network for further testing?

What is your environment?

hello Angela Dai.
I am interested in your work.
I try to run your code,but an error occurs.
But an error occurs

running install_lib
running build_py
running build_ext
building 'sparseconvnet.SCN' extension
/usr/local/cuda/bin/nvcc -I/home/lin/voxel/SparseConvNet/sparseconvnet/SCN/ -I/home/lin/miniconda3/envs/cgnn/lib/python2.7/site-packages/torch/include -I/home/lin/miniconda3/envs/cgnn/lib/python2.7/site-packages/torch/include/torch/csrc/api/include -I/home/lin/miniconda3/envs/cgnn/lib/python2.7/site-packages/torch/include/TH -I/home/lin/miniconda3/envs/cgnn/lib/python2.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/lin/miniconda3/envs/cgnn/include/python2.7 -c sparseconvnet/SCN/cuda.cu -o build/temp.linux-x86_64-2.7/sparseconvnet/SCN/cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -std=c++14 -Xcompiler -fopenmp -O3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=SCN -D_GLIBCXX_USE_CXX11_ABI=0
nvcc fatal : Value 'c++14' is not defined for option 'std'
error: command u'/usr/local/cuda/bin/nvcc' failed with exit status 1

my current nvcc is for CUDA8 and ubuntu 16.04.
what is your environment?

Usage of 'GenerateScans'

Hello @angeladai ,

Thanks for sharing!

We are very interested in your project，and we want to try it on our own dataset. I noticed that you've provided the 'GenrateScans' to create the .sdf files which suits for the project. But we ran into the same problem(#1) while implementing. So could you please explain a bit more about the usage of 'GenerateScans', for example the inputs' format and their meanings. Thanks in advance!

Bests,
Bob

Failed to download data.

Excuse me, but is the data servers still online?

marching_cubes_cpp import error

I have run sudo python2 setup.py install under torch/marching_cubes/ to install marching_cubes.
Then, I run python2 test_scene.py --gpu 0 --input_data_path ./data/mp_sdf_vox_2cm_input.zip --target_data_path ./data/mp_sdf_vox_2cm_target.zip --test_file_list ../filelists/mp-rooms_val-scenes.txt --model_path sgnn.pth --output --max_to_vis 20, error occurs:

Traceback (most recent call last):
  File "test_scene.py", line 11, in <module>
    import data_util
  File "/home/user01/ZhengJiafang/sgnn/torch/data_util.py", line 9, in <module>
    import marching_cubes.marching_cubes as mc
  File "/home/user01/ZhengJiafang/sgnn/torch/marching_cubes/marching_cubes.py", line 6, in <module>
    import marching_cubes_cpp
  File "build/bdist.linux-x86_64/egg/marching_cubes_cpp.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/marching_cubes_cpp.py", line 6, in __bootstrap__
ImportError: /home/user01/.cache/Python-Eggs/marching_cubes_cpp-0.0.0-py2.7-linux-x86_64.egg-tmp/marching_cubes_cpp.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

How can I fix this?

the mean of your database

I want to understand the mean of your database.
I try your code.

I see input data is a dictionary format.
like this
input_data{'name','input','sdf','world2grid','known','hierarchy','orig_dims'}.
I think the input_data['input'][0] is some point cloud.
I want to understand the mean of ['input'][1]

I think the input_data['input'] can be get by reshaping input_data['sdf'].
but i don't what is 'sdf'.

I want to try my point cloud or .ply with your model to reconstruction.
Please explain to me, thank you!

Testing on my own dataset.

Hi @angeladai Thanks a lot for your sharing.

I would like to try your model on my own dataset which is also an indoor scene dataset. But I meet some problems during preparing the input since I do not have a Windows to use the code in GenerateScans. So I use the code from here to compute the TSDF volume and change the load_scene function accordingly to test on my own data.

I visualize the generated input mesh and predicted mesh and find that: 1. the input mesh looks quite different (much more sparse) from the mesh directly extract from the TSDF volume using marching cube. 2. the predicted mesh contains a big plane where it is empty in the input mesh. I suspect this situation is because my input preparation might have some mistake.

Could you provide some help? Thanks in advance.