We provide code for the reproduction of the main results in Jointly Learning Visual and Auditory Speech Representations from Raw Data. Our implementation is based on PyTorch Lightning.

conda env create -f environment.yml. Change the environment prefix to match the location of miniconda3, if necessary.

1. The datasets used in the paper can be downloaded from the following links:
   • LRS3
   • VoxCeleb2
   • LRS2
2. Compute 68 landmarks per frame using e.g., RetinaFace and 2-D FAN, or download them e.g., from this repo. Each landmark file should have the same name as its corresponding video (except that it ends in .npy).
3. Use the following command to crop the mouths:

   python preprocessing/extract_mouths.py --src_dir ${SOURCE_DIR} --tgt_dir ${TARGET_DIR} --landmarks_dir ${LANDMARKS_DIR}
   

Below are the checkpoints of the Base and Large models pre-trained with RAVEn on LRS3+Vox2-en.

• Below are the checkpoints corresponding to Tables 1 and 2 for VSR and ASR on LRS3. Models are provided for both low- and high-resource labelled data settings. In the high-resource setting, the models are fine-tuned on the full LRS3 dataset (433 hours). In the low-resource setting, they are fine-tuned on a subset ("trainval") of LRS3 (30 hours).

• In some cases, the models were re-trained so the WER may differ slightly from the ones shown in the paper (which are also reproduced below).

• The paths for the slurm bash scripts used for inference are shown in the table below. Note that the scripts may need to be modified according to the cluster environment.

• The language model we used in this work can be found here.

Code for pre-training and fine-tuning coming soon...

If you find this repo useful for your research, please consider citing the following:

@article{haliassos2022jointly,
  title={Jointly Learning Visual and Auditory Speech Representations from Raw Data},
  author={Haliassos, Alexandros and Ma, Pingchuan and Mira, Rodrigo and Petridis, Stavros and Pantic, Maja},
  journal={arXiv preprint arXiv:2212.06246},
  year={2022}
}