Xingguang Zhong* ยท Yue Pan* ยท Jens Behley ยท Cyrill Stachniss
(* Equal Contribution)
| Incremental Mapping | Reconstruction Results |
|---|---|
shine_incremental.mp4 |
shine_reconresult.mp4 |
Table of Contents
Accurate mapping of large-scale environments is an essential building block of most outdoor autonomous systems. Challenges of traditional mapping methods include the balance between memory consumption and mapping accuracy. This paper addresses the problems of achieving large-scale 3D reconstructions with implicit representations using 3D LiDAR measurements. We learn and store implicit features through an octree-based hierarchical structure, which is sparse and extensible. The features can be turned into signed distance values through a shallow neural network. We leverage binary cross entropy loss to optimize the local features with the 3D measurements as supervision. Based on our implicit representation, we design an incremental mapping system with regularization to tackle the issue of catastrophic forgetting in continual learning. Our experiments show that our 3D reconstructions are more accurate, complete, and memory-efficient than current state-of-the-art 3D mapping methods.
git clone [email protected]:PRBonn/SHINE_mapping.git
cd SHINE_mapping
conda create --name shine python=3.7
conda activate shine
Kaolin depends on Pytorch (>= 1.8, <= 1.13.1), please install the corresponding Pytorch for your CUDA version (can be checked by nvcc --version). You can find the installation commands here.
For example, for CUDA version >=11.6, you can use:
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
Kaolin now supports installation with wheels. For example, to install kaolin 0.12.0 over torch 1.12.1 and cuda 11.6:
pip install kaolin==0.12.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-1.12.1_cu116.html
[Or you can build kaolin by yourself (click to expand)]
Follow the instructions to install kaolin. Firstly, clone kaolin to a local directory:
git clone --recursive https://github.com/NVIDIAGameWorks/kaolin
cd kaolin
Then install kaolin by:
python setup.py develop
Use python -c "import kaolin; print(kaolin.__version__)" to check if kaolin is successfully installed.
pip install open3d scikit-image wandb tqdm natsort pyquaternion
Note that you CUDA version must be >=11.6.2 to be compatible with the container.
https://docs.docker.com/engine/install/ubuntu/
https://developer.nvidia.com/nvidia-container-runtime
git clone [email protected]:PRBonn/SHINE_mapping.git
cd SHINE_mapping
docker build --tag shine .
mkdir /tmp/shine_test_data
docker run --rm -v ${pwd}/:/repository -v /tmp/shine_test_data:/data -it --gpus all shine
Results will be produced in /tmp/shine_test_data/results.
docker run --rm -v .:/repository -v ${MY_DATA_DIR}:/data -it --gpus all shine bash
where ${MY_DATA_DIR} is the directory on the host with data in the format described in config/kitti/docker_kitti_batch.yaml.
Once inside the container RUN as described below. Results will be found on host in ${MY_DATA_DIR}/results .
Generally speaking, you only need to provide:
pc_path: the folder containing the point cloud (.bin,.plyor.pcdformat) for each frame.pose_path: the pose file (.txt) containing the transformation matrix of each frame.calib_path: the calib file (.txt) containing the static transformation between sensor and body frames (optional, would be identity matrix if set as'').
They all follow the KITTI odometry data format.
After preparing the data, you need to correctly set the data path (pc_path, pose_path and calib_path) in the config files under config folder. You may also set a path output_root to store the experiment results and logs.
Here, we provide the link to several publicly available datasets for testing SHINE Mapping:
Download the dataset from here or use the following script to download (3.4GB):
sh ./scripts/download_maicity.sh
Download the full dataset from here.
If you want to use an example part of the dataset (seq 00) for the test, you can use the following script to download (117 MB):
sh ./scripts/download_kitti_example.sh
Download the full dataset from here.
If you want to use an example part of the dataset (Quad) for the test, you can use the following script to download (634 MB):
sh ./scripts/download_ncd_example.sh
SHINE Mapping also supports the mapping on RGB-D datasets. You may first try the synthetic dataset from NeuralRGB-D. You can download the full dataset from here or use the following script to download (7.25 GB).
sh ./scripts/download_neural_rgbd_data.sh
After downloading the data, you need to convert the dataset to the KITTI format by using for each sequence:
sh ./scripts/convert_rgbd_to_kitti_format.sh
[Details (click to expand)]
Our method is currently a mapping-with-known-pose system. If you do not have the ground truth pose file, you may use a LiDAR odometry system such as KISS-ICP to easily estimate the pose.
You can simply install KISS-ICP by:
pip install kiss-icp
And then run KISS-ICP with your data path pc_path
kiss_icp_pipeline <pc_path>
The estimated pose file can be found in ./results/latest/velodyne.txt. You can directly use it as your pose_path. In this case, you do not need a calib file, so just set calib_path: "" in the config file.
Check the repository Color-SHINE-MAPPING (thanks @ZorAttC for the contribution) for using SHINE Mapping to reconstruct colorized mesh using colorized point cloud.
We take the MaiCity dataset as an example to show how SHINE Mapping works. You can simply replace maicity with your dataset name in the config file path, such as ./config/[dataset]/[dataset]_[xxx].yaml.
The results will be stored with your experiment name with the starting timestamp in the output_root directory as what you set in the config file. You can find the reconstructed mesh (*.ply format) and optimized model in mesh and model folder, respectively. If the save_map option is turned on, then you can find the grid sdf map in map folder.
For mapping based on offline batch processing, use:
python shine_batch.py ./config/maicity/maicity_batch.yaml
[Expected results (click to expand)]
maicity_shine_batch_20cm.mp4
For incremental mapping with replay strategy (within a local bounding box), use:
python shine_incre.py ./config/maicity/maicity_incre_replay.yaml
An interactive visualizer would pop up if you set o3d_vis_on: True (by default) in the config file. You can press space to pause and resume.
[Expected results (click to expand)]
For the sake of efficiency, we sacrifice a bit mapping quality to use a 50cm leaf voxel size for the feature octree here.
To only visualize the mesh in a local bounding box for faster operation, you can set mc_local: True and mc_with_octree: False in the config file.
maicity_shine_incre_replay_50cm.mp4
For incremental mapping with a regularization strategy, use:
python shine_incre.py ./config/maicity/maicity_incre_reg.yaml
[Expected results (click to expand)]
For the sake of efficiency, we sacrifice a bit of mapping quality to use a 50cm leaf voxel size for the feature octree.
maicity_shine_incre_reg_50cm.mp4
[Expected results on other datasets (click to expand)]
KITTI
Newer College
Apollo
Wild Place Forests
IPB Office
Replica
ICL Living Room
The logs can be monitored via Weights & Bias online if you turn the wandb_vis_on option on. If it's your first time to use Weights & Bias, you would be requested to register and login to your wandb account.
To evaluate the reconstruction quality, you need to provide the (reference) ground truth point cloud and your reconstructed mesh. The ground truth point cloud can be found (or sampled from) the downloaded folder of MaiCity, Newer College and Neural RGBD datasets.
Please change the data path and evaluation set-up in ./eval/evaluator.py and then run:
python ./eval/evaluator.py
to get the reconstruction metrics such as Chamfer distance, completeness, F-score, etc.
As mentioned in the paper, we also compute a fairer accuracy metric using the ground truth point cloud masked by the intersection of the reconstructed meshes of all the compared methods. To generate such masked ground truth point clouds, you can configure the data path in ./eval/crop_intersection.py and then run it.
To reproduce the quantitative results on MaiCity and Newer College dataset in the paper, you can use the config files in ./config/config_icra2023/. The reconstructed meshes can also be downloaded from here. Note that these numbers are obtained using the batch mapping mode. You can achieve similar results using the incremental mapping mode with the replay strategy.
[Details (click to expand)]
-
You can play with different loss functions for SHINE Mapping. With the
ray_loss: Falseoption, the loss would be calculated from the sdf at each sample point. In this case, you can then select fromsdf_bce(the proposed method),sdf_l1andsdf_l2loss as themain_loss_type. With theray_loss: Trueoption, the loss would be calculated from each ray containing multiple point samples as a depth rendering procedure. In this case, you can select fromdranddr_neusas themain_loss_type. According to our experiments, using our proposedsdf_bceloss can achieve the best reconstruction efficiently. We can get a decent reconstruction of a scene with several hundred frames in just one minute. Additionally, you can use theekional_loss_onoption to turn on/off the Ekional loss and useweight_eas its weight. -
The feature octree is built mainly according to
leaf_vox_size,tree_level_worldandtree_level_feat.leaf_vox_sizerepresents the size of the leaf voxel size in meter.tree_level_worldandtree_level_featrepresent the total tree level and the tree levels with latent feature codes, respectively.tree_level_worldshould be large enough to guarantee all the map data lies inside the cube with the sizeleaf_vox_size**(tree_level_world+1). -
SHINE Mapping supports both offline batch mapping and incremental sequential mapping. For incremental mapping, one can either load a fixed pre-trained decoder from the batching mapping on a similar dataset (set
load_model: True) or train the decoder forfreeze_after_frameframes on-the-fly and then freeze it afterwards (setload_model: False). The first option would lead to better mapping performance. -
You can use the
mc_vis_levelparameter to have a trade-off between the scene completion and the exact measurement accuracy. This parameter indicates at which level of the octree the marching cubes reconstruction would be conducted. The larger the value ofmc_vis_level(but not larger thantree_level_feat), the more scene completion ability you would gain (but also some artifacts such as a double wall may appear). And with the small value, SHINE mapping would only reconstruct the part with actual measurements without filling the holes. The safest way to avoid the holes on the ground is to setmc_mask_on: Falseto disable the masking for marching cubes. By turning on themc_with_octreeoption, you can achieve a faster marching cubes reconstruction only in the region within the octree nodes. -
The incremental mapping with regularization strategy (setting
continual_learning_reg: True) can achieve incremental neural mapping without storing an ever-growing data pool which would be a burden for the memory. The coefficientlambda_forgetneeds to be fine-tuned under different feature octree and point sampling settings. The recommended value is from1e5to1e8. A pre-trained decoder is also recommended to be loaded during incremental mapping with regularization for better performance. -
We also provide an option to conduct incremental mapping with a replay strategy in a local sliding window. You can turn this on by setting
window_replay_on: Truewith a validwindow_radius_mindicating the size of the sliding window. -
It's also possible to incorporate semantic information in our SHINE-Mapping framework. You may set
semantic_on = Truein theutils/config.pyfile to enable semantic mapping and also provide semantic supervision by setting thelabel_pathin the config file. The labels should be in Semantic KITTI format. An example semantic reconstruction results using Semantic KITTI can be downloaded from here.
If you use SHINE Mapping for any academic work, please cite our original paper.
@inproceedings{zhong2023icra,
title={SHINE-Mapping: Large-Scale 3D Mapping Using Sparse Hierarchical Implicit NEural Representations},
author={Zhong, Xingguang and Pan, Yue and Behley, Jens and Stachniss, Cyrill},
booktitle = {Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)},
year={2023}
}
If you have any questions, please contact:
- Xingguang Zhong {[email protected]}
- Yue Pan {[email protected]}
This work has partially been funded by the European Unionโs HORIZON programme under grant agreement No 101070405 (DigiForest) and grant agreement No 101017008 (Harmony).
Additionally, we thank greatly for the authors of the following opensource projects:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent