This repo is the OpenGprahAU tool.
demo:
Models were traiend on hybrid dataset of 2,000k images.
This hybrid dataset includes:
The tool can predict action units of 41 categories:
| AU1 | AU2 | AU4 | AU5 | AU6 | AU7 | AU9 | AU10 | AU11 | AU12 | AU13 | AU14 | AU15 | AU16 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Inner brow raiser | Outer brow raiser | Brow lowerer | Upper lid raiser | Cheek raiser | Lid tightener | Nose wrinkler | Upper lip raiser | Nasolabial deepener | Lip corner puller | Sharp lip puller | Dimpler | Lip corner depressor | Lower lip depressor |
| AU17 | AU18 | AU19 | AU20 | AU22 | AU23 | AU24 | AU25 | AU26 | AU27 | AU32 | AU38 | AU39 | - |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Chin raiser | Lip pucker | Tongue show | Lip stretcher | Lip funneler | Lip tightener | Lip pressor | Lips part | Jaw drop | Mouth stretch | Lip bite | Nostril dilator | Nostril compressor | - |
| AUL1 | AUL1 | AUL2 | AUR2 | AUL4 | AUR4 | AUL6 | AUR6 | AUL10 | AUR10 | AUL12 | AUR12 | AUL14 | AUR14 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Left inner brow raiser | Right inner brow raiser | Left outer brow raiser | Right outer brow raiser | Left brow lowerer | Right brow lowerer | Left cheek raiser | Right cheek raiser | Left upper lip raiser | Right upper lip raiser | Left nasolabial deepener | Right nasolabial deepener | Left dimpler | Right dimpler |
We provide tools for prepareing data in tool/.
After Downloading raw data files, you can use these tools to process them, aligning with our protocals.
We divide the dataset into three independent parts (i.e., train, val, test).
Hybrid Dataset
| arch_type | GoogleDrive link | Average F1-score | Average Acc. |
|---|---|---|---|
Ours (MobileNetV3) |
- | - | - |
Ours (ResNet-18) |
link | 22.33 | 92.97 |
Ours (ResNet-50) |
link | 22.52 | 92.63 |
Ours (Swin-Tiny) |
link | 22.66 | 92.97 |
Ours (Swin-Small) |
link | 24.49 | 92.84 |
Ours (Swin-Base) |
link | 23.53 | 92.91 |
| arch_type | GoogleDrive link | Average F1-score | Average Acc. |
|---|---|---|---|
Ours (MobileNetV3) |
- | - | - |
Ours (ResNet-18) |
link | 22.51 | 93.23 |
Ours (ResNet-50) |
link | 23.24 | 93.31 |
Ours (Swin-Tiny) |
link | 22.74 | 93.37 |
Ours (Swin-Small) |
- | - | - |
Ours (Swin-Base) |
- | - | - |
- to detect facial action units in a facial image using our stage1 model, run:
python demo.py --arc resnet50 --stage 1 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth --input demo_imgs/1014.jpg --draw_text
- to detect facial action units in a facial image using our stage2 model, run:
python demo.py --arc resnet50 --stage 2 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_second_stage.pth --input demo_imgs/1014.jpg --draw_text
- to train the first stage of our approach (ResNet-50) on hybrid Dataset, run:
python train_stage1.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_first_stage -b 512 -lr 0.00002
- to test the first stage of our approach (SwinT) on hybrid Dataset, run:
python test_stage1.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_first_stage --resume ./results/OpenGprahAU-SwinT_first_stage/bs_64_seed_0_lr_2e-05/best_model.pth
- to train the second stage of our approach (ResNet-50) on hybrid Dataset, run:
python train_stage2.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_second_stage -b 512 -lr 0.00001 --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth
- to test the second stage of our approach (SwinT) on hybrid Dataset, run:
python test_stage2.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_second_stage --resume ./results/OpenGprahAU-SwinT_second_stage/bs_64_seed_0_lr_1e-05/best_model.pth
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent