[CoRL 2023] BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities

Abstract: Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios.

• 2024/04/29 Support BM2CP in OPV2V dataset.
• 2024/03/12 Support How2comm(NeurIPS2023).
• 2024/01/28 Support OPV2V dataset. Support SCOPE(ICCV2023). Fix bugs in fusion dataset that cause abnormal performance in presence of pose error.
• 2023/11/30 First version.

• Dataset Support

  • OPV2V
  • V2XSet
  • V2X-Sim 2.0
  • DAIR-V2X

• Spconv Support

  • 1.2.1
  • 2.x

• SOTA collaborative perception method support

  • Late Fusion
  • Early Fusion
  • When2com (CVPR2020)
  • V2VNet (ECCV2020)
  • DiscoNet (NeurIPS2021)
  • V2X-ViT (ECCV2022)
  • CoBEVT (CoRL2022)
  • Where2comm (NeurIPS2022)
  • CoAlign (ICRA2023)
  • BM2CP (CoRL2023)
  • SCOPE (ICCV2023)
  • How2comm (NeurIPS2023)

• Visualization

  • BEV visualization
  • 3D visualization

Download raw data of OPV2V relseased by Official.

1. Download raw data of DAIR-V2X.
2. Download complemented annotation from Yifan Lu.

Please refer to the INSTALL.md for detailed documentations.

First of all, modify the dataset path in the setting file, i.e. xxx.yaml.

data_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure"
root_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure/train.json"
validate_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure/val.json"
test_dir: "{YOUR PATH}/DAIR-V2X-C/cooperative-vehicle-infrastructure/val.json"

The setting is same as OpenCOOD, which uses yaml file to configure all the parameters for training. To train your own model from scratch or a continued checkpoint, run the following commonds:

python opencood/tools/train.py --hypes_yaml ${CONFIG_FILE} [--model_dir  ${CHECKPOINT_FOLDER}]

Arguments Explanation:

• hypes_yaml: the path of the training configuration file, e.g. opencood/hypes_yaml/second_early_fusion.yaml, meaning you want to train an early fusion model which utilizes SECOND as the backbone. See Tutorial 1: Config System to learn more about the rules of the yaml files.
• model_dir (optional) : the path of the checkpoints. This is used to fine-tune the trained models. When the model_dir is given, the trainer will discard the hypes_yaml and load the config.yaml in the checkpoint folder.

For example, to train BM2CP from scratch:

python opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/dair-v2x/dair_bm2cp.yaml

To train BM2CP from a checkpoint:

python opencood/tools/train.py --hypes_yaml opencood/hypes_yaml/dair-v2x/dair_bm2cp.yaml --model_dir opencood/logs/dair_bm2cp_2023_11_28_08_52_46

Before you run the following command, first make sure the validation_dir in config.yaml under your checkpoint folder refers to the testing dataset path, e.g. opv2v_data_dumping/test.

python opencood/tools/inference.py --model_dir ${CHECKPOINT_FOLDER} --fusion_method ${FUSION_STRATEGY} --eval_epoch ${epoch_number} --save_vis ${default False}

Arguments Explanation:

• model_dir: the path to your saved model.
• fusion_method: indicate the fusion strategy, currently support 'early', 'late', 'intermediate', 'no'(indicate no fusion, single agent), 'intermediate_with_comm'(adopt intermediate fusion and output the communication cost).
• eval_epoch: int. Choose to inferece which epoch.
• save_vis: bool. Wether to save the visualization result.

The evaluation results will be dumped in the model directory.

Thank for the excellent cooperative perception codebases OpenCOOD, CoPerception and Where2comm.

Thank for the excellent cooperative perception datasets DAIR-V2X, OPV2V and V2X-SIM.

Thank for the dataset and code support by DerrickXu, Yue Hu and YiFan Lu.

Thanks for the insightful previous works in cooperative perception field.

V2VNet: Vehicle-to-vehicle communication for joint perception and prediction ECCV20 [Paper]

When2com: Multi-agent perception via communication graph grouping CVPR20 [Paper] [Code]

Learning Distilled Collaboration Graph for Multi-Agent Perception NeurIPS21 [Paper] [Code]

V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer ECCV2022 [Paper] [Code] [Talk]

Self-Supervised Collaborative Scene Completion: Towards Task-Agnostic Multi-Robot Perception CoRL2022 [Paper]

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers CoRL2022 [Paper] [Code]

Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps NeurIPS2022 [Paper] [Code]

Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception ICCV2023 [Paper][Code]

How2comm: Communication-Efficient and Collaboration-Pragmatic Multi-Agent Perceptio NeurIPS2023 [Paper] [Code]

OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication ICRA2022 [Paper] [Website] [Code]

V2X-Sim: A Virtual Collaborative Perception Dataset and Benchmark for Autonomous Driving RAL21 [Paper] [Website][Code]

DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection CVPR2022 [Paper] [Website] [Code]