ldkong1205 / pointcloud-c Goto Github PK

View Code? Open in Web Editor NEW

164.0 25.0 22.0 34.62 MB

Benchmarking and Analyzing Point Cloud Perception Robustness under Corruptions

Home Page: https://pointcloud-c.github.io/home

Python 86.80% Shell 0.90% Makefile 0.02% C++ 5.71% Cuda 6.12% CMake 0.03% C 0.41%

3dvision benchmark corruption pointcloud robustness

pointcloud-c's Introduction

Benchmarking and Analyzing Point Cloud Perception Robustness under Corruptions

Jiawei Ren, Lingdong Kong, Liang Pan, Ziwei Liu
S-Lab, Nanyang Technological University

About

PointCloud-C is the very first test-suite for point cloud perception robustness analysis under corruptions. It includes two sets: ModelNet-C (ICML'22) for point cloud classification and ShapeNet-C (arXiv'22) for part segmentation.

Fig. Examples of point cloud corruptions in PointCloud-C.

Visit our project page to explore more details. 🌱

Updates

[2024.03] - We add Leaderboard to this page. We welcome pull requests to submit your results!
[2024.01] - The toolkit tailored for The RoboDrive Challenge has been released. 🛠️
[2023.12] - We are hosting The RoboDrive Challenge at ICRA 2024. 🚙
[2023.03] - Intend to test the robustness of your 3D perception models on real-world point clouds? Check our recent work, Robo3D, a comprehensive suite that enables OoD robustness evaluation of 3D detectors and segmentors on our newly established datasets: KITTI-C, SemanticKITTI-C, nuScenes-C, and WOD-C.
[2022.11] - The preprint of the PointCloud-C paper (ModelNet-C + ShapeNet-C) is available here.
[2022.10] - We have successfully hosted the 2022 PointCloud-C Challenge. Congratulations to the winners: 🥇 Antins_cv, 🥈 DGPC & DGPS, and 🥉 BIT_gdxy_xtf.
[2022.07] - Try a Gradio demo for PointCloud-C corruptions at Hugging Face Spaces! 🤗
[2022.07] - Competition starts! Join now at our CodaLab page.
[2022.06] - PointCloud-C is now live on Paper-with-Code. Join the benchmark today!
[2022.06] - The 1st PointCloud-C challenge will be hosted in conjecture with the ECCV'22 SenseHuman workshop. 🚀
[2022.06] - We are organizing the 1st PointCloud-C challenge! Click here to explore the competition details.
[2022.05] - ModelNet-C is accepted to ICML 2022. Click here to check it out! 🎉

Overview

Highlight
Data Preparation
Getting Started
Leaderboard
Benchmark Results
Evaluation
Customize Evaluation
Build PointCloud-C
TODO List
License
Acknowledgement
Citation

Highlight

Corruption Taxonomy

ModelNet-C (Classification)

ShapeNet-C (Part Segmentation)

Data Preparation

Please refer to DATA_PREPARE.md for the details to prepare the ModelNet-C and ShapeNet-C datasets.

Getting Started

Please refer to GET_STARTED.md to learn more usage about this codebase.

Leaderboard

Method	Reference	Augmentation	mCE $\downarrow$	Clean OA $\uparrow$
EPiC (RPC, WOLFMix)	Levi et al., ICCV 2023	Yes	0.501	0.927
EPiC (PCT)	Levi et al., ICCV 2023	No	0.646	0.934
WOLFMix (GDANet)	Ren et al., ICML 2022	Yes	0.571	0.934
RPC	Ren et al., ICML 2022	No	0.863	0.930

Benchmark Results

ModelNet-C (Classification)

Method	Reference	Standalone	mCE $\downarrow$	RmCE $\downarrow$	Clean OA $\uparrow$
DGCNN	Wang et al.	Yes	1.000	1.000	0.926
PointNet	Qi et al.	Yes	1.422	1.488	0.907
PointNet++	Qi et al.	Yes	1.072	1.114	0.930
RSCNN	Liu et al.	Yes	1.130	1.201	0.923
SimpleView	Goyal et al.	Yes	1.047	1.181	0.939
GDANet	Xu et al.	Yes	0.892	0.865	0.934
CurveNet	Xiang et al.	Yes	0.927	0.978	0.938
PAConv	Xu et al.	Yes	1.104	1.211	0.936
PCT	Guo et al.	Yes	0.925	0.884	0.930
RPC	Ren et al.	Yes	0.863	0.778	0.930
OcCo (DGCNN)	Wang et al.	No	1.248	1.262	0.922
PointBERT	Yu et al.	No	1.033	0.895	0.922
PointMixUp (PointNet++)	Chen et al.	No	1.028	0.785	0.915
PointCutMix-K (PointNet++)	Zhang et al.	No	0.806	0.808	0.933
PointCutMix-R (PointNet++)	Zhang et al.	No	0.796	0.809	0.929
PointWOLF (DGCNN)	Kim et al.	No	0.814	0.698	0.926
RSMix (DGCNN)	Lee et al.	No	0.745	0.839	0.930
PointCutMix-R (DGCNN)	Zhang et al.	No	0.627	0.504	0.926
PointCutMix-K (DGCNN)	Zhang et al.	No	0.659	0.585	0.932
WOLFMix (DGCNN)	Ren et al.	No	0.590	0.485	0.932
WOLFMix (GDANet)	Ren et al.	No	0.571	0.439	0.934
WOLFMix (PCT)	Ren et al.	No	0.574	0.653	0.934
PointCutMix-K (PCT)	Zhang et al.	No	0.644	0.565	0.931
PointCutMix-R (PCT)	Zhang et al.	No	0.608	0.518	0.928
WOLFMix (RPC)	Ren et al.	No	0.601	0.940	0.933

ShapeNet-C (Part Segmentation)

Method	Reference	Standalone	mCE $\downarrow$	RmCE $\downarrow$	Clean mIoU $\uparrow$
DGCNN	Wang et al.	Yes	1.000	1.000	0.852
PointNet	Qi et al.	Yes	1.178	1.056	0.833
PointNet++	Qi et al.	Yes	1.112	1.850	0.857
OcCo-DGCNN	Wang et al.	No	0.977	0.804	0.851
OcCo-PointNet	Wang et al.	No	1.130	0.937	0.832
OcCo-PCN	Wang et al.	No	1.173	0.882	0.815
GDANet	Xu et al.	Yes	0.923	0.785	0.857
PAConv	Xu et al.	Yes	0.927	0.848	0.859
PointTransformers	Zhao et al.	Yes	1.049	0.933	0.840
PointMLP	Ma et al.	Yes	0.977	0.810	0.853
PointBERT	Yu et al.	No	1.033	0.895	0.855
PointMAE	Pang et al.	No	0.927	0.703	0.860

*Note: Standalone indicates whether or not the method is a standalone architecture or a combination with augmentation or pretrain.

Evaluation

Evaluation commands are provided in EVALUATE.md.

Customize Evaluation

We have provided evaluation utilities to help you evaluate on ModelNet-C using your own codebase. Please follow CUSTOMIZE.md.

Build PointCloud-C

You can manage to generate your own "PointCloud-C"! Follow the instructions in GENERATE.md.

TODO List

Initial release. 🚀
Add license. See here for more details.
Release test sets. Download ModelNet-C and ShapeNet-C from our project page.
Add evaluation scripts for classification models.
Add evaluation scripts for part segmentation models.
Add competition details.
Clean and retouch codebase.

License

This work is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Acknowledgement

We acknowledge the use of the following public resources during the course of this work: ¹SimpleView, ²PCT, ³GDANet, ⁴CurveNet, ⁵PAConv, ⁶RSMix, ⁷PointMixUp, ⁸PointCutMix, ⁹PointWOLF, ¹⁰PointTransformers, ¹¹OcCo, ¹²PointMLP, ¹³PointBERT, and ¹⁴PointMAE.

Citation

If you find this work helpful, please kindly consider citing our papers:

@article{ren2022pointcloud-c,
  title = {PointCloud-C: Benchmarking and Analyzing Point Cloud Perception Robustness under Corruptions},
  author = {Jiawei Ren and Lingdong Kong and Liang Pan and Ziwei Liu},
  journal = {Preprint},
  year = {2022}
}

@inproceedings{ren2022modelnet-c,
  title = {Benchmarking and Analyzing Point Cloud Classification under Corruptions},
  author = {Jiawei Ren and Liang Pan and Ziwei Liu},
  booktitle = {International Conference on Machine Learning (ICML)},
  year = {2022}
}

pointcloud-c's People

Contributors

Stargazers

Watchers

pointcloud-c's Issues

Question about pretraining-based methods.

Whether corruptions that are part of ModelNet-C and ShapeNet-C are allowed in the pretraining phase?

As we know these methods (e.g. OcCo, Point-BERT & Point-MAE) all use similar corruptions like drop local in unlabeled data.

about the code of WOLFMix

Hi @jiawei-ren , i have searched the whole file for codes of WOLFMix but failed, i only found the codes of RSMix and PointWOLF. Would you please release the code of WOLFMix and update corresponding training codes, i think this will be great value to helping understanding the workflow.

Questions about datasets and augmentations.

Hello ldkong1205:

Thanks for the PointCloud-C Benchmark and Challenge for point cloud analysis.

I have two questions for the Challenge on the Codalab, mainly about datasets and augmentations.

Is the full PointCloud-C dataset (downloaded from https://pointcloud-c.github.io/download.html) allowed for model training ?
"Corruptions that are part of ModelNet-C and ShapeNet-C are strictly NOT allowed to be included as training augmentations" means we can't use the 7 augmentations (listed at https://pointcloud-c.github.io/details.html) during training ? However, some augmentations, such as jitter and scale, are widely used training tricks in previous works.

Looking forward to your reply.

Is preatrained model for PCT+WolfMix missing?

Hi, I couldn't find PCT+WolfMix weights. Please let me know where I can find it. Thanks.

Can't reproduce RPC + WolfMix results with the provided weights.

Hi, The RPC + WolfMix results with your pre-trained model differ from the paper's results. Any idea why? Thanks.

python PCT/main.py --exp_name=test --num_points=1024 --use_sgd=True --eval_corrupt=True --model_path pretrained_models/RPC_WOLFMix_final.t7 --test_batch_size 8 --model RPC

Namespace(batch_size=32, beta=0.0, dataset='modelnet40', dropout=0.5, epochs=250, eval=False, eval_corrupt=True, exp_name='test', jitter=False, knn=False, lr=0.0001, model='RPC', model_path='pretrained_models/RPC_WOLFMix_final.t7', momentum=0.9, no_cuda=False, nsample=512, num_points=1024, pw=False, rddrop=False, rdscale=False, rot=False, rsmix_prob=0.5, seed=1, shift=False, shuffle=False, test_batch_size=8, use_sgd=True, w_R_range=10, w_S_range=3, w_T_range=0.25, w_num_anchor=4, w_sample_type='fps', w_sigma=0.5)
Using GPU : 0 from 1 devices

{'acc': 0.923419773095624, 'avg_per_class_acc': 0.8932906976744185, 'corruption': 'clean'}
{'OA': 0.923, 'corruption': 'clean', 'level': 'Overall'}
{'acc': 0.9116693679092382, 'avg_per_class_acc': 0.8796686046511628, 'corruption': 'scale', 'level': 0}
{'acc': 0.9136952998379254, 'avg_per_class_acc': 0.8816279069767441, 'corruption': 'scale', 'level': 1}
{'acc': 0.906807131280389, 'avg_per_class_acc': 0.8727558139534886, 'corruption': 'scale', 'level': 2}
{'acc': 0.9116693679092382, 'avg_per_class_acc': 0.8772965116279069, 'corruption': 'scale', 'level': 3}
{'acc': 0.9120745542949756, 'avg_per_class_acc': 0.87625, 'corruption': 'scale', 'level': 4}
{'CE': 0.947, 'OA': 0.911, 'RCE': 0.6, 'corruption': 'scale', 'level': 'Overall'}
{'acc': 0.9173419773095624, 'avg_per_class_acc': 0.8749941860465116, 'corruption': 'jitter', 'level': 0}
{'acc': 0.8865478119935171, 'avg_per_class_acc': 0.8260523255813954, 'corruption': 'jitter', 'level': 1}
{'acc': 0.8103727714748784, 'avg_per_class_acc': 0.7123197674418604, 'corruption': 'jitter', 'level': 2}
{'acc': 0.6511345218800648, 'avg_per_class_acc': 0.524656976744186, 'corruption': 'jitter', 'level': 3}
{'acc': 0.4266612641815235, 'avg_per_class_acc': 0.32084883720930235, 'corruption': 'jitter', 'level': 4}
{'CE': 0.829, 'OA': 0.738, 'RCE': 0.764, 'corruption': 'jitter', 'level': 'Overall'}
{'acc': 0.9238249594813615, 'avg_per_class_acc': 0.8925813953488373, 'corruption': 'rotate', 'level': 0}
{'acc': 0.9217990275526742, 'avg_per_class_acc': 0.8923313953488371, 'corruption': 'rotate', 'level': 1}
{'acc': 0.923419773095624, 'avg_per_class_acc': 0.8891627906976745, 'corruption': 'rotate', 'level': 2}
{'acc': 0.8987034035656402, 'avg_per_class_acc': 0.8598837209302325, 'corruption': 'rotate', 'level': 3}
{'acc': 0.8456239870340356, 'avg_per_class_acc': 0.8082790697674419, 'corruption': 'rotate', 'level': 4}
{'CE': 0.451, 'OA': 0.903, 'RCE': 0.142, 'corruption': 'rotate', 'level': 'Overall'}
{'acc': 0.9153160453808752, 'avg_per_class_acc': 0.8827558139534883, 'corruption': 'dropout_global', 'level': 0}
{'acc': 0.9124797406807131, 'avg_per_class_acc': 0.8762558139534884, 'corruption': 'dropout_global', 'level': 1}
{'acc': 0.9055915721231766, 'avg_per_class_acc': 0.8673430232558139, 'corruption': 'dropout_global', 'level': 2}
{'acc': 0.8804700162074555, 'avg_per_class_acc': 0.834029069767442, 'corruption': 'dropout_global', 'level': 3}
{'acc': 0.7467585089141004, 'avg_per_class_acc': 0.7035290697674418, 'corruption': 'dropout_global', 'level': 4}
{'CE': 0.516, 'OA': 0.872, 'RCE': 0.293, 'corruption': 'dropout_global', 'level': 'Overall'}
{'acc': 0.9076175040518638, 'avg_per_class_acc': 0.8779244186046512, 'corruption': 'dropout_local', 'level': 0}
{'acc': 0.8772285251215559, 'avg_per_class_acc': 0.8394709302325583, 'corruption': 'dropout_local', 'level': 1}
{'acc': 0.8286061588330632, 'avg_per_class_acc': 0.7695232558139534, 'corruption': 'dropout_local', 'level': 2}
{'acc': 0.7726904376012966, 'avg_per_class_acc': 0.7092906976744185, 'corruption': 'dropout_local', 'level': 3}
{'acc': 0.6717990275526742, 'avg_per_class_acc': 0.6149418604651162, 'corruption': 'dropout_local', 'level': 4}
{'CE': 0.908, 'OA': 0.812, 'RCE': 0.835, 'corruption': 'dropout_local', 'level': 'Overall'}
{'acc': 0.8905996758508914, 'avg_per_class_acc': 0.8544593023255814, 'corruption': 'add_global', 'level': 0}
{'acc': 0.8472447325769854, 'avg_per_class_acc': 0.7881860465116277, 'corruption': 'add_global', 'level': 1}
{'acc': 0.8107779578606159, 'avg_per_class_acc': 0.7417383720930232, 'corruption': 'add_global', 'level': 2}
{'acc': 0.7548622366288493, 'avg_per_class_acc': 0.6675813953488372, 'corruption': 'add_global', 'level': 3}
{'acc': 0.6977309562398704, 'avg_per_class_acc': 0.6104883720930232, 'corruption': 'add_global', 'level': 4}
{'CE': 0.678, 'OA': 0.8, 'RCE': 0.557, 'corruption': 'add_global', 'level': 'Overall'}
{'acc': 0.8614262560777958, 'avg_per_class_acc': 0.803970930232558, 'corruption': 'add_local', 'level': 0}
{'acc': 0.8399513776337115, 'avg_per_class_acc': 0.776389534883721, 'corruption': 'add_local', 'level': 1}
{'acc': 0.813614262560778, 'avg_per_class_acc': 0.7350639534883721, 'corruption': 'add_local', 'level': 2}
{'acc': 0.7876823338735819, 'avg_per_class_acc': 0.7037848837209302, 'corruption': 'add_local', 'level': 3}
{'acc': 0.7759319286871961, 'avg_per_class_acc': 0.6841220930232559, 'corruption': 'add_local', 'level': 4}
{'CE': 0.669, 'OA': 0.816, 'RCE': 0.532, 'corruption': 'add_local', 'level': 'Overall'}
{'RmCE': 0.532, 'mCE': 0.714, 'mOA': 0.836}

link broken

Hi, @ldkong1205 ,

The link [ShapeNet-C]https://github.com/ldkong1205/PointCloud-C/blob/main is broken. Would you help to fix it?

Thanks~

I'm sorry, but where is the implementation of the proposed RPC network?

Greetings!
Really solid work. However, I cannot find codes about the RPC network as presented in Section 6. Do I miss it?

about the shape of competition dataset(cls_extra_test_data.h)

hi @ldkong1205 , i find the shape of the data in cls_extra_test_data.h is (24680, 724, 3). In dim=1, the number of points in point cloud(724), it's not a normal number(1024 or more) of point cloud, and 724 is not the same as the number of points in modelnet-c dataset (2468, 1024, 3). This may cause some confusion in training process(use 1024 points for training or 724 points).

An extra class in part segmentation

Hi!

Thank you for the nice work. I wanted to ask that why do the corruptions in PartSegmentation dataset have an extra category. The original 'number of parts' in the partnet 'clean dataset' are 50 (i.e. [0-49]).

However, I see that in the corruptions generated for PartNet dataset, there are 51 categories for parts. Can you please tell me what is the extra category?

Best,
Mirza.

about git+git://github.com/imankgoyal/[email protected]#egg=etw_pytorch_utils

hi, thanks for your work. My server cant connect to github, so i cant download this on shell. Besides, i have never seen command like 'git+git:', so i dont know how to find alternative ways. Would you be kind to tell me alternative ways?

Thanks!

Data Augmentation in PCT

In the PCT main script (line), it says 'implement augmentation'. Do I need to add code here for data augmentation? Where do I find the augmentation code?

In your paper, you mentioned, you used scaling, rotation, and translation during training for PCT + WOLFMix and RPC+WOLDMix, is this correct? or you used more augmentation.

Thank you.

Question about the ShapeNet-C paper.

Hi, Thank you for the great work.

While I am using your Dataset (ModelNet-C and ShapeNet-C), I cannot find the related paper for ShapeNet-C (arxiv 22).
Could you share it so that I can cite the paper?

Thanks

ldkong1205 / pointcloud-c Goto Github PK

pointcloud-c's Introduction

Benchmarking and Analyzing Point Cloud Perception Robustness under Corruptions

About

Updates

Overview

Highlight

Corruption Taxonomy

ModelNet-C (Classification)

ShapeNet-C (Part Segmentation)

Data Preparation

Getting Started

Leaderboard

Benchmark Results

ModelNet-C (Classification)

ShapeNet-C (Part Segmentation)

Evaluation

Customize Evaluation

Build PointCloud-C

TODO List

License

Acknowledgement

Citation

pointcloud-c's People

Contributors

Stargazers

Watchers

Forkers

pointcloud-c's Issues

Recommend Projects

Recommend Topics

Recommend Org