• Introduction
• Benchmark
• Dataset Preparation
• How to test
• How to train
• Dependencies
• Citation
• License

This is the repository of Learn an Effective Lip Reading Model without Pains. In this repository, we provide pre-trained models and training settings for deep lip reading. We evaluate our pipeline on LRW Dataset and LRW1000 Dataset. We obtain 88.4% and 56.0% on LRW and LRW-1000, respectively. The results are comparable and even surpass current state-of-the-art results. Especially, we reach the current state-of-the-art result (56.0%) on LRW-1000 Dataset.

1. Download LRW Dataset and LRW1000 Dataset and link lrw_mp4 and LRW1000_Public in the root of this repository:

ln -s PATH_TO_DATA/lrw_mp4 .
ln -s PATH_TO_DATA/LRW1000_Public .

2. You can run scripts/prepare_lrw.py and scripts/prepare_lrw1000.py to generate training samples of LRW and LRW-1000 Dataset respectively:

python scripts/prepare_lrw.py
python scripts/prepare_lrw1000.py 

The mouth videos, labels, and word boundary information will be saved in the .pkl format. We pack image sequence as jpeg format into our .pkl files and decoding via PyTurboJPEG. You may need to modify the utils/dataset.py file when training your own dataset.

Link of pretrained weights: Baidu Yun (code: 26qn), Google Drive

If you can not access to provided links, please email [email protected] or [email protected].

To test our provided weights, you should download weights and place them in the root of this repository.

For example, to test baseline on LRW Dataset:

python main_visual.py \
    --gpus='0'  \
    --lr=0.0 \
    --batch_size=128 \
    --num_workers=8 \
    --max_epoch=120 \
    --test=True \
    --save_prefix='checkpoints/lrw-baseline/' \
    --n_class=500 \
    --dataset='lrw' \
    --border=False \
    --mixup=False \
    --label_smooth=False \
    --se=False \
    --weights='checkpoints/lrw-cosine-lr-acc-0.85080.pt'

To test our model in LRW-1000 Dataset:

python main_visual.py \
    --gpus='0'  \
    --lr=0.0 \
    --batch_size=128 \
    --num_workers=8 \
    --max_epoch=120 \
    --test=True \
    --save_prefix='checkpoints/lrw-1000-final/' \
    --n_class=1000 \
    --dataset='lrw1000' \
    --border=True \
    --mixup=False \
    --label_smooth=False \
    --se=True \
    --weights='checkpoints/lrw1000-border-se-mixup-label-smooth-cosine-lr-wd-1e-4-acc-0.56023.pt'

For example, to train lrw baseline:

python main_visual.py \
    --gpus='0,1,2,3'  \
    --lr=3e-4 \
    --batch_size=400 \
    --num_workers=8 \
    --max_epoch=120 \
    --test=False \
    --save_prefix='checkpoints/lrw-baseline/' \
    --n_class=500 \
    --dataset='lrw' \
    --border=False \
    --mixup=False \
    --label_smooth=False \
    --se=False  

Optional arguments:

• gpus: the GPU id used for training
• lr: learning rate. By default, we automatically applied the Linear Scale Rule in code (e.g., lr=3e-4 for 4 GPUs x 32 video/gpu and lr=1.2e-3 for 8 GPUs x 128 video/gpu). We recommend lr=3e-4 for 32 video/gpu when training from scratch. You need to modify the learning rate based on your setting.
• batch_size: batch size
• num_workers: the number of processes used for data loading
• max_epoch: the maximum epochs in training
• test: The test mode. When using this mode, the program will only test once and exit.
• weights(optional): The path of pre-trained weight. If this option is specified, the model will load the pre-trained weights by the given location.
• save_prefix: the save prefix of model parameters
• n_class: the number of total word classes
• dataset: the dataset used for training and testing, only lrw and lrw1000 are supported.
• border: use word boundary indicated variable for training and testing
• mixup: use mixup in training
• label_smooth: use label_smooth in training
• se: use se module in ResNet-18

More training details and setting can be found in our paper. We plan to include more pretrained models in the future.

• PyTorch 1.6
• opencv-python
• TurboJPEG and PyTurboJPEG

If you find this code useful in your research, please consider to cite the following papers:

@article{feng2020learn,
  author       = "Feng, Dalu and Yang, Shuang and Shan, Shiguang and Chen, Xilin",
  title        = "Learn an Effective Lip Reading Model without Pains",
  journal      = "arXiv preprint arXiv:2011.07557",
  year         = "2020",
}

The MIT License