This repository provides a robustness evaluation on Face Recognition by using various adversarial attacks. These evaluations are conducted under diverse adversarial settings, incuding doding and impersonation attacks, and
attacks, white-box and black-box attacks. More details and some findings can be reffered to the manuscript.
- Introduction
- Installation
- Documentation
- Data Preparation
- White-Box Benchmark
- Black-Box Benchmark
- Black-Box API Benchmark
- Acknowledgements
- Citation
- This repository studies various backbones (e.g., ResNet, IR, MobileNet, ShuffleNet, etc.) and various losses (e.g., Softmax, SphereFace, CosFace, ArcFace, etc.). Some trained models and source codes are provided.
- This repository introduces various white-box attacks including FGSM, BIM, MIM, CW, and black-box attack methods including FGSM, BIM, MIM, CIM and LGC, Evolutionary, etc.
- This repository aims to help researchers understand the adversarial robustness and provide a reliable evaluate criteria for robustness of the future works on face recognition.
- Our paper also provides some valuable insights for the design of more robust models in facial tasks, as well as in other metric learning tasks such as image retrieval, person re-identification, etc.
- RobFR takes an easily extendable implementation for every module due to independent interface, thus enabling more researchers to conveniently supplement new contents.
- git clone https://github.com/ShawnXYang/Face-Robustness-Benchmark
- pip install -e .
- pip install -r requirements.txt
We provide API docs at https://face-robustness-benchmark.readthedocs.io/.
We build our work on previously released data, and support the following datasets (continuously updating): LFW, YTF, CFP-FP, MegaFace.
LFW
Put LFW dataset and `pairs.txt` to `data`.data
|---lfw
| |
| |---AJ_Cook
| | |
| | |---AJ_Cook_0001.jpg
| |
| |---xxxx
| | |
...........
|---pairs.txt
The pairs.txt
can be seen in here.
Then you can execute scripts/align_image_lfw.py
to build aligned versions of LFW dataset(multiple resolutions).
data
|---lfw
|---lfw-112x112
|---lfw-160x160
|---lfw-112x96
|---pairs.txt
YTF
Similarily, the file structure will be as follows:
data
|---splits.txt
|---ytf-112x112
|---ytf-160x160
|---ytf-112x96
CFP
data
|---cfp-112x112
|---cfp-160x160
|---cfp-112x96
- The median distance of the minimum perturbations against dodging and impersonation attacks under the
and
norms.
Clean | FGSM | FGSM | BIM | BIM | MIM | MIM | CW | CW | FGSM | FGSM | BIM | BIM | MIM | MIM | |
dod. | imp. | dod. | imp. | dod. | imp. | dod. | imp. | dod. | imp. | dod. | imp. | dod. | imp. | ||
FaceNet | 99.2 | 0.92 | 1.17 | 0.59 | 0.66 | 0.66 | 0.73 | 0.54 | 0.61 | 1.75 | 2.14 | 1.17 | 1.31 | 1.28 | 1.42 |
SphereFace | 98.2 | 0.73 | 0.64 | 0.58 | 0.52 | 0.62 | 0.55 | 0.56 | 0.50 | 1.31 | 1.11 | 1.06 | 0.92 | 1.12 | 0.97 |
CosFace | 98.7 | 0.97 | 0.73 | 0.69 | 0.56 | 0.75 | 0.59 | 0.65 | 0.54 | 1.69 | 1.25 | 1.30 | 1.00 | 1.39 | 1.06 |
ArcFace | 99.5 | 1.09 | 0.80 | 0.83 | 0.66 | 0.89 | 0.69 | 0.79 | 0.64 | 1.97 | 1.42 | 1.53 | 1.20 | 1.62 | 1.25 |
MobileFace | 99.5 | 1.11 | 0.62 | 0.75 | 0.50 | 0.83 | 0.53 | 0.71 | 0.49 | 1.98 | 1.12 | 1.44 | 0.94 | 1.55 | 0.97 |
MobileNet | 99.4 | 1.03 | 0.64 | 0.67 | 0.47 | 0.73 | 0.50 | 0.62 | 0.44 | 1.75 | 1.08 | 1.22 | 0.83 | 1.31 | 0.88 |
MobileNetV2 | 99.3 | 0.89 | 0.64 | 0.62 | 0.50 | 0.69 | 0.53 | 0.59 | 0.48 | 1.55 | 1.12 | 1.17 | 0.91 | 1.25 | 0.95 |
ShuffleNetV1 | 99.5 | 1.12 | 0.53 | 0.67 | 0.41 | 0.75 | 0.44 | 0.62 | 0.39 | 2.02 | 0.97 | 1.28 | 0.77 | 1.39 | 0.80 |
ShuffleNetV2 | 99.2 | 1.06 | 0.62 | 0.66 | 0.45 | 0.72 | 0.48 | 0.61 | 0.43 | 1.86 | 1.08 | 1.23 | 0.84 | 1.31 | 0.88 |
ResNet50 | 99.7 | 1.53 | 0.84 | 0.86 | 0.58 | 0.97 | 0.62 | 0.79 | 0.55 | 2.64 | 1.44 | 1.59 | 1.05 | 1.75 | 1.11 |
IR | 99.6 | 1.30 | 1.05 | 0.84 | 0.73 | 0.94 | 0.81 | 0.78 | 0.71 | 2.36 | 1.86 | 1.66 | 1.41 | 1.80 | 1.50 |
JPEG | 99.6 | 2.46 | 2.00 | 1.36 | 1.16 | 1.31 | 1.12 | 0.75 | 0.67 | 4.19 | 3.31 | 2.77 | 2.23 | 2.58 | 2.16 |
Bit-Red | 99.6 | 1.37 | 1.06 | 0.86 | 0.73 | 0.92 | 0.78 | 0.78 | 0.74 | 3.00 | 2.00 | 2.00 | 1.12 | 2.00 | 1.67 |
R|P | 99.4 | 4.50 | 8.01 | 2.05 | 2.20 | 2.28 | 2.50 | 1.86 | 2.02 | 8.00 | 10.31 | 4.56 | 4.77 | 4.25 | 4.59 |
PGD-AT | 91.3 | 4.14 | 3.95 | 2.37 | 2.37 | 2.70 | 2.70 | 2.06 | 2.14 | 12.94 | 12.38 | 10.34 | 10.23 | 10.83 | 10.62 |
TRADES | 91.0 | 4.37 | 4.41 | 2.70 | 2.73 | 3.03 | 3.03 | 2.38 | 2.51 | 12.69 | 12.12 | 10.59 | 10.30 | 10.97 | 10.64 |
- The attack success rate vs. perturbation budget curves of the models against dodging attacks under the
norm.
- The attack success rate vs. perturbation budget curves of the models against impersonation attacks under the
norm.
run_white.sh
provides some command line interfaces to run white-box evaluation. For example, run FGSM evaluation on MobileFace for LFW dataset using distance as:
python benchmark/lfw/FGSM_white.py --distance=l2 --goal=dodging --model=MobileFace --eps=16 --log=log-lfw-FGSM-l2-dodging-MobileFace-white.txt
Then the attack results are saved in --log
.
adv_img,tar_img,score,dist,success
1.npy,data/lfw-112x112/Abel_Pacheco/Abel_Pacheco_0004.jpg,0.21092090010643005,1.0467989629677874,1
2.npy,data/lfw-112x112/Akhmed_Zakayev/Akhmed_Zakayev_0003.jpg,0.21074934303760529,4.202811928700617,1
3.npy,data/lfw-112x112/Akhmed_Zakayev/Akhmed_Zakayev_0003.jpg,0.21039743721485138,2.1047161963395666,1
4.npy,data/lfw-112x112/Amber_Tamblyn/Amber_Tamblyn_0002.jpg,0.20931993424892426,1.2771732226518993,1
....
score
indicates the similarity predicted by victim model, dist
means the minimal adversarial or
distortion distance, and
success
means whether this attack is successful.
run_black.sh
provides some command line interfaces to run white-box evaluation.
# generate adversarial examples
python benchmark/lfw/FGSM_black.py --distance=l2 --goal=dodging --model=MobileFace --eps=4 --output=outputs/lfw-FGSM-l2-dodging-MobileFace --batch_size=20
# generate log file
python benchmark/lfw/run_test.py --model=Mobilenet --distance=l2 --anno=outputs/lfw-FGSM-l2-dodging-MobileFace/annotation.txt --log=log-lfw-Mobilenet-FGSM-l2-dodging-MobileFace-black.txt --goal=dodging
After executing the first script, the adversarial examples are saved as png files in --output
. An annotation file (annotation.txt
) is also saved in --output
.
Then run_test.py
will generate the evaluation log file in --log
, and the format of the log file is same as log file of White-Box Evaluation.
- For the training procedure of Face Recognition, we mainly refer to the public code from face.evoLVe.PyTorch.
- To benchmark adversarial robustness on image classification, we recommend RealSafe, a Python library for adversarial machine learning research.
If you benefit from our work in your research, please consider to cite the following paper:
@article{yang2020delving,
title={Delving into the Adversarial Robustness on Face Recognition},
author={Yang, Xiao and Yang, Dingcheng and Dong, Yinpeng and Yu, Wenjian and Su, Hang and Zhu, Jun},
journal={arXiv preprint arXiv:2007.04118},
year={2020}
}