SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

Abstract

In this paper, we introduce SalsaNext for the uncertainty-aware semantic segmentation of a full 3D LiDAR point cloud in real-time. SalsaNext is the next version of SalsaNet [1] which has an encoder-decoder architecture where the encoder unit has a set of ResNet blocks and the decoder part combines upsampled features from the residual blocks. In contrast to SalsaNet, we introduce a new context module, replace the ResNet encoder blocks with a new residual dilated convolution stack with gradually increasing receptive fields and add the pixel-shuffle layer in the decoder. Additionally, we switch from stride convolution to average pooling and also apply central dropout treatment. To directly optimize the Jaccard index, we further combine the weighted cross-entropy loss with Lovasz-Softmax loss [2]. We finally inject a Bayesian treatment to compute the epistemic and aleatoric uncertainties for each point in the cloud. We provide a thorough quantitative evaluation on the Semantic-KITTI dataset [3], which demonstrates that the proposed SalsaNext outperforms other state-of-the-art semantic segmentation.

Examples

Video

Semantic Kitti Segmentation Scores

How to use the code

First create the anaconda env with: conda env create -f salsanext.yml then activate the environment with conda activate salsanext.

To train/eval you can use the following scripts:

• Training script (you might need to chmod +x the file)
  • We have the following options:
    • -d [String] : Path to the dataset
    • -a [String]: Path to the Architecture configuration file
    • -l [String]: Path to the main log folder
    • -n [String]: additional name for the experiment
    • -c [String]: GPUs to use (default no gpu)
    • -u [String]: If you want to train an Uncertainty version of SalsaNext (default false)
  • For example if you have the dataset at /dataset the architecture config file in /salsanext.yml and you want to save your logs to /logs to train "salsanext" with 2 GPUs with id 3 and 4:
    • ./train.sh -d /dataset -a /salsanext.yml -m salsanext -l /logs -c 3,4

• Eval script (you might need to chmod +x the file)
  • We have the following options:
    • -d [String]: Path to the dataset
    • -p [String]: Path to save label predictions
    • -m [String]: Path to the location of saved model
    • -s [String]: Eval on Validation or Train (standard eval on both separately)
    • -u [String]: If you want to infer using an Uncertainty model (default false)
    • -c [Int]: Number of MC sampling to do (default 30)
  • If you want to infer&evaluate a model that you saved to /salsanext/logs/[the desired run] and you want to infer$eval only the validation and save the label prediction to /pred:
    • ./eval.sh -d /dataset -p /pred -m /salsanext/logs/[the desired run] -s validation -n salsanext

Disclamer

We based our code on RangeNet, please go show some support!

Citation

@misc{cortinhal2020salsanext,
    title={SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving},
    author={Tiago Cortinhal and George Tzelepis and Eren Erdal Aksoy},
    year={2020},
    eprint={2003.03653},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

References

[1] E. E. Aksoy, S. Baci, and S. Cavdar, “Salsanet: Fast road and vehicle segmentation in lidar point clouds for autonomous driving,” in IEEE Intelligent Vehicles Symposium (IV2020), 2020.

[2] M. Berman, A. Rannen Triki, and M. B. Blaschko, “The lova ́sz- softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks,” in CVPR, 2018.

[3] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stach- niss, and J. Gall, “SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences,” in ICCV, 2019.

[4] A. Kendall, V. Badrinarayanan, and R. Cipolla, “Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding,” arXiv preprint arXiv:1511.02680, 2015.

[5] R. P. K. Poudel, S. Liwicki, and R. Cipolla, “Fast-scnn: Fast semantic segmentation network,” CoRR, vol. abs/1902.04502, 2019.

[6] B. Wu, A. Wan, X. Yue, and K. Keutzer, “Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud,” ICRA, 2018.

[7] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, “RangeNet++: Fast and Accurate LiDAR Semantic Segmentation,” in IROS, 2019.

[8] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, “Deep learning for 3d point clouds: A survey,” CoRR, 2019.

[9] W. Shi, J. Caballero, F. Husza ́r, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super- resolution using an efficient sub-pixel convolutional neural network,” CoRR, vol. abs/1609.05158, 2016.

[10] B. Wu, X. Zhou, S. Zhao, X. Yue, and K. Keutzer, “Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud,” in ICRA, 2019.

[11] Y. Wang, T. Shi, P. Yun, L. Tai, and M. Liu, “Pointseg: Real-time semantic segmentation based on 3d lidar point cloud,” CoRR, 2018.

[12] E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation.” PAMI, 2016.

[13] C. Zhang, W. Luo, and R. Urtasun, “Efficient convolutions for real- time semantic segmentation of 3d point clouds,” in 3DV, 2018.

[14] O. Ronneberger, P.Fischer, and T. Brox, “U-net: Convolutional net- works for biomedical image segmentation,” in Medical Image Com- puting and Computer-Assisted Intervention, 2015, pp. 234–241.

[15] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in CVPR, 2017. [16] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in NIPS, 2017.

[17] L. Landrieu and M. Simonovsky, “Large-scale point cloud semantic segmentation with superpoint graphs,” in CVPR, 2018.

[18] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum pointnets for 3d object detection from RGB-D data,” CoRR, 2017.

[19] Y. Zhou and O. Tuzel, “Voxelnet: End-to-end learning for point cloud based 3d object detection,” in CVPR, 2018.

[20] L. P. Tchapmi, C. B. Choy, I. Armeni, J. Gwak, and S. Savarese, “Segcloud: Semantic segmentation of 3d point clouds,” in IEEE Intl. Conf. on 3D Vision (3DV), 2017, p. 537–547.

[21] F. J. Lawin, M. Danelljan, P. Tosteberg, G. Bhat, F. S. Khan, and M. Felsberg, “Deep projective 3d semantic segmentation,” CoRR, 2017. [Online]. Available: http://arxiv.org/abs/1705.03428

[22] H. Su, V. Jampani, D. Sun, S. Maji, E. Kalogerakis, M. Yang, and J. Kautz, “Splatnet: Sparse lattice networks for point cloud processing,” in CVPR, 2018.

[23] R.AlexandruRosu,P.Schu ̈tt,J.Quenzel,andS.Behnke,“LatticeNet: Fast Point Cloud Segmentation Using Permutohedral Lattices,” arXiv e-prints, p. arXiv:1912.05905, Dec. 2019.

[24] C. Xu, B. Wu, Z. Wang, W. Zhan, P. Vajda, K. Keutzer, and M. Tomizuka, “Squeezesegv3: Spatially-adaptive convolution for effi- cient point-cloud segmentation,” 2020.

[25] Y. Zeng, Y. Hu, S. Liu, J. Ye, Y. Han, X. Li, and N. Sun, “Rt3d: Real-time 3-d vehicle detection in lidar point cloud for autonomous driving,” IEEE RAL, vol. 3, no. 4, pp. 3434–3440, Oct 2018.

[26] M. Simon, S. Milz, K. Amende, and H. Gross, “Complex-yolo: Real- time 3d object detection on point clouds,” CoRR, 2018.

[27] I.Alonso,L.Riazuelo,L.Montesano,andA.C.Murillo,“3d-mininet: Learning a 2d representation from point clouds for fast and efficient 3d lidar semantic segmentation,” 2020.

[28] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning, 2016, pp. 1050–1059.

[29] D.Feng,L.Rosenbaum,andK.Dietmayer,“Towardssafeautonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection,” in ITSC. IEEE, 2018, pp. 3266–3273.

[30] E. Ilg, O. Cicek, S. Galesso, A. Klein, O. Makansi, F. Hutter, and T. Brox, “Uncertainty estimates and multi-hypotheses networks for optical flow,” in ECCV, 2018, pp. 652–667.

[31] B. Zhang and P. Wonka, “Point cloud instance segmentation using probabilistic embeddings,” CoRR, 2019.

[32] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770–778.

[33] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolu- tional networks,” CoRR, vol. abs/1311.2901, 2013.

[34] X. Li, S. Chen, X. Hu, and J. Yang, “Understanding the disharmony between dropout and batch normalization by variance shift,” arXiv preprint arXiv:1801.05134, 2018.

[35] Y. Gal, J. Hron, and A. Kendall, “Concrete dropout,” in Advances in Neural Information Processing Systems, 2017, pp. 3581–3590.

[36] M. Tatarchenko, J. Park, V. Koltun, and Q. Zhou, “Tangent convolu- tions for dense prediction in 3d,” in CVPR, 2018.

[37] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, “Randla-net: Efficient semantic segmentation of large- scale point clouds,” 2019.

[38] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in CVPR, 2012.