Project Page | Video | Paper
This repository contains the implementation of our paper:
InsMOS: Instance-Aware Moving Object Segmentation in LiDAR Data
Neng Wang, Chenghao Shi, Ruibin Guo, Huimin Lu, Zhiqiang Zheng, Xieyuanli Chen
-
Our instance-aware moving object segmentation on the SemanticKITTI sequence 08 and 20, 21.
-
Red points indicate predicted moving points, cyan indicate predicted static instance points and gray points are static background.
-
Green bounding boxes represent cars, blue bounding boxes represent pedestrians, and yellow bounding boxes represent cyclists.
- [2023/8/12] Code released!
- [2023/6/22] Our work is accepted for IROS2023 π
1γSemanticKITTI: Download SemanticKITTI dataset from the official website.
2γKITTI-road Dataset: Download the KITTI-road Velodyne point clouds from the official website and MOS label from MotionSeg3D.
3γInstance label: Download the box labels from ondrive or BaiduDisk,code:59t7, and please refer to boundingbox_label_readme about more details of instance label .
Data structure
βββ sequences
βββ 00/
β βββ velodyne/
| | βββ 000000.bin
| | βββ 000001.bin
| | βββ ...
β βββ labels/
| | βββ 000000.label
| | βββ 000001.label
| | βββ ...
| βββ boundingbox_label
| | βββ 000000.npy
| | βββ 000001.npy
| | βββ ...
| βββ calib.txt
| βββ poses.txt
| βββ times.txt
βββ 01/ # 00-10 for training
βββ 08/ # for validation
βββ 11-21/ # 11-21 for testing
# kitti-road
βββ 30 31 32 33 34 40 # for training
βββ 35 36 37 38 39 41 # for testing
system dependencies:
ubuntu20.04, CUDA 11.3, cuDNN 8.2.1,
python dependencies:
python 3.7
conda create --name insmos python=3.7
conda activate insmos
pip install -r requirements.txt
# insltall pytorch with cuda11.3, avoid using "pip install torch"
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
# ensure numpy==1.18.1
pip uninstall numpy
pip install numpy==1.18.1
Install MinkowskiEngine :
cd ~
mkdir ThirdParty
sudo apt-get install libopenblas-dev
git clone https://github.com/NVIDIA/MinkowskiEngine.git
cd MinkowskiEngine
conda activate insmos
python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --blas=openblas
# clone code
git clone https://github.com/nubot-nudt/InsMOS.git
cd InsMOS
# activate conda
conda activate insmos
# install
python setup.py develop
Run the following command to evaluate the model in SemanticKITTI validation dataset or test dataset. At this moment, a βpreb_outβ folder will be generated, which contains βbbox_prebβ, βconfidenceβ, and βmos_prebβ for storing the predicted bounding boxes, confidence scores for moving points, and labels for moving points, respectively.
We public the model was trained on the SemanticKITTI dataset (N_10_t_0.1_odom.ckpt) and the other model was trained on the Semantic-KITTI and KITTI-road dataset (N_10_t_0.1_odom_road.ckpt). You can download from ondrive or BaiduDisk,code:59t7, and then put the model in "ckpt" folder.
cd InsMOS
python scripts/predict_mos.py --cfg_file config/config.yaml --data_path /path/to/kitti/sequences --ckpt ./ckpt/N_10_t_0.1_odom.ckpt --split valid
We use the semantickitti-api to evaluate the MOS IOU.
cd semantic-kitti-api
python evaluate_mos.py --dataset /path/to/kitti --predictions ./preb_out/InsMOS/mos_preb --split valid
Run the following command to refine the network predictions.
python scripts/refine.py --data_path /path/to/kitti/sequences --split valid
Re-evaluate the results of refinement.
python evaluate_mos.py --dataset /path/to/kitti --predictions ./preb_out_refine/mos_preb --split valid
Run the following command to visualize the results of moving object segmentation and instance prediction.
Press key n
to show next frame.
Press key b
to show last frame.
Press key q
to quit display.
cd visual
python vis_mos_bbox.py
You can set single gpu or multi gpu for training in train.py. We set batch size to 4 for each gpu. During the training process, there may be an increase in GPU memory consumption, so it is advisable not to set the batch size too large initially. We test 4-6 is fine on 3090 GPU.
export DATA=/path/to/kitti/sequences
python scripts/train.py
If the training process is interrupted unexpectedly, you can resume the training using the following command.
python scripts/train.py --weights ./logs/InsMOS/version_x/checkpoints/last.ckpt --checkpoint ./logs/InsMOS/version_x/checkpoints/last.ckpt
If you use our code in your work, please star our repo and cite our paper.
@article{wang2023arxiv,
title={{InsMOS: Instance-Aware Moving Object Segmentation in LiDAR Data}},
author={Wang, Neng and Shi, Chenghao and Guo, Ruibin and Lu, Huimin and Zheng, Zhiqiang and Chen, Xieyuanli},
journal={arXiv preprint},
doi = {10.48550/ARXIV.2303.03909},
volume = {2303.03909},
year={2023}
}
Any question or suggestions are welcome!
Neng Wang: [email protected] and Xieyuanli Chen: [email protected]
This project is free software made available under the MIT License. For details see the LICENSE file.