Chenfeng Xu, Bichen Wu, Ji Hou, Sam Tsai, Ruilong Li, Jialiang Wang, Wei Zhan, Zijian He, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka

[paper], [project page]

NeRF-Det is a novel method for 3D detection with posed RGB images as input. Our method makes novel use of NeRF in an end-to-end manner to explicitly estimate 3D geometry, thereby improving 3D detection performance. Specifically, to avoid the significant extra latency associated with per-scene optimization of NeRF, we introduce sufficient geometry priors to enhance the generalizability of NeRF-MLP. We subtly connect the detection and NeRF branches through a shared MLP, enabling an efficient adaptation of NeRF to detection and yielding geometry-aware volumetric representations for 3D detection. As a result of our joint-training design, NeRF-Det is able to generalize well to unseen scenes for object detection, view synthesis, and depth estimation tasks without per-scene optimization.

Inspired by our previous work SOLOFusion, we further advance our method with using long-frame-low-res images, which is more efficient and effective. We provide the checkpoints and configs with better performance below.

If you find this work helpful, please cite:

Citation:

@inproceedings{
  xu2023nerfdet,
  title={NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection},
  author={Xu, Chenfeng and Wu, Bichen and Hou, Ji and Tsai, Sam and Li, Ruilong and Wang, Jialiang and Zhan, Wei and He, Zijian and Vajda, Peter and Keutzer, Kurt and Tomizuka, Masayoshi},
  booktitle={ICCV},
  year={2023},
}

@inproceedings{
park2023time,
title={Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection},
author={Jinhyung Park and Chenfeng Xu and Shijia Yang and Kurt Keutzer and Kris M. Kitani and Masayoshi Tomizuka and Wei Zhan},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=H3HcEJA2Um}
}

For convenience, we provide a Dockerfile. Alternatively, you can install all required packages manually.

This implementation is based on mmdetection3d framework and ImVoxelNet. Please refer to the original installation guide install.md, replacing open-mmlab/mmdetection3d with saic-vul/imvoxelnet. Also, rotated_iou should be installed with these 4 commands.

Most of the NeRF-Det-related code locates in the following files: detectors/nerfdet.py, model_utils/render_ray.py, model_utils/projection.py, pipelines/multi_view.py, pipelines/formating.py.

We provide the code of ScanNet experiments. Please follow instructions in scannet to prepare the dataset.

Please see getting_started.md for basic usage examples.

Test pre-trained model using dist_test with nerfdet configs:

bash tools/dist_test.sh configs/nerfdet/nerfdet_res50_2x_low_res.py \
    work_dirs/nerfdet_res50_2x_low_res/res50_2x_low_res.pth 4 --eval mAP

If you want to render novel view, you can modify let "render_testing = True" in config. You can then get results like below.

if you want to have the bounding box visualization, you can run

python tools/test.py configs/nerfdet/nerfdet_res50_2x_low_res.py \
    work_dirs/nerfdet_res50_2x_low_res/latest.pth --show \
    --show-dir work_dirs/nerfdet_res50_2x_low_res

For better visualizations, you may set score_thr in configs to 0.15 or more:

You can train your own model by the training script

bash tools/dist_train.sh configs/nerfdet/nerfdet_res50_2x_low_res.py 4

https://github.com/SamsungLabs/imvoxelnet

https://ibrnet.github.io

https://github.com/open-mmlab/mmdetection3d

The majority of Nerf-Det is licensed under CC-BY-NC, however portions of the project are available under separate license terms: https://github.com/SamsungLabs/imvoxelnet is licensed under the MIT license.