This repository provides a leaderboard for certifiable robustness against adversarial patch attacks.

Check out our paper list and tutorials for adversarial patch attacks and defenses!

For now, this leaderboard focuses on the setting of

1. Image classification task
   • For Object Detection, see the ObjectSeeker paper.
   • For Semantic Segmentation, see this paper.
2. ImageNet dataset
3. One 2% square patch anywhere on the image
4. Prediction Recovery vs. attack detection (read this explanation for the difference between these two notions)

Stay tuned for future updates on different datasets and tasks :)

We include the following properties of different defenses in this leaderboard.

1. Defense name and publication venue.
2. Certified robust accuracy: certified robust accuracy for a 2%-pixel square patch anywhere on the image. All leaderboards are sorted by certified robust accuracy
3. Clean accuracy: clean accuracy of the defended models
4. Vanilla clean accuracy: clean accuracy of vanilla undefended models (if the defense is built upon off-the-shelf models)
5. Backbone: the backbone used for each defense
6. Comment: additional comments for each defense, including SOTA results, defense parameters, special training recipes.
7. Code: most defenses have source code implemented by the authors or others

Note: We discuss more properties of different defenses in another copy of leaderboards hosted on Google Sheet.

If you have new results and want to contribute to the leaderboard. Please submit a pull request or email me (Chong Xiang; [email protected])

• Prediction Recovery
• Attack Detection

The prediction recovery defense requires the defense model to always predict/recover the robust label (without any abstention).

(go back to table of contents)

The attack detection defense allows the defense to alert and abstain from making predictions when it detects an attack.

1. Attack detection defenses usually can tune the defense parameters to balance the trade-off between clean accuracy and robust accuracy.
2. Here, I only report three representative points for each defense setup (different confidence threshold tau)
   • low-tau: high robust accuracy and low clean accuracy; clean accuracy and robust accuracy are the same
   • high-tau: low robust accuracy and high clean accuracy; clean accuracy is close to vanilla undefended clean accuracy
   • mid-tau: in between
3. You are encouraged to play with the code yourself :)

1. The original Minority Reports paper does not discuss high-resolution images like ImageNet.
2. The results here (Minority Reports + PatchCleanser) are from the discussion section of the PatchCleanser paper.
3. the idea is to plug PatchCleanser's mask generation approach into the MR defense design https://github.com/inspire-group/PatchCleanser/blob/main/misc/pc_mr.py

(go back to table of contents)