- Reproduce various works based on unified standards ๐
- High-speed training and testing โก
- Easy to extend ๐ญ
-
Clone or download this repository.
-
Install the required packages:
pytorch>=1.8.1 wandb>=0.12.10
-
Download the dataset:
The dataset is based on VoxCeleb and is divided into train/valid/test sets according to "Learnable Pins: Crossmodal Embeddings for Person Identity, 2018, ECCV" (901/100/250).
Download
dataset.zip
from Google Drive (2.3GB) and unzip it to the project root directory. The folder structure should be as follows:dataset โโโ evals โ โโโ test_matching_10.pkl โ โโโ test_matching_g.pkl โ โโโ test_matching.pkl โ โโโ test_retrieval.pkl โ โโโ test_verification_g.pkl โ โโโ test_verification.pkl โ โโโ valid_verification.pkl โโโ info โ โโโ name2gender.pkl โ โโโ name2jpgs_wavs.pkl โ โโโ name2movies.pkl โ โโโ name2voice_id.pkl โ โโโ train_valid_test_names.pkl โ โโโ works โ โโโ wen_weights.txt โโโ face_input.pkl โโโ voice_input.pkl
-
Learnable Pins: Crossmodal Embeddings for Person Identity, 2018, ECCV
python works/1_pins.py
-
Face-Voice Matching using Cross-modal Embeddings, MM, 2018
python works/2_FV-CME.py
-
On Learning Associations of Faces and Voices, ACCV, 2018
python works/3_LAFV.py
-
Disjoint Mapping Network for Cross-modal Matching of Voices and Faces, ICLR, 2019
python works/11_SS_DIM_VFMR_Barlow.py --name=DIMNet
-
Voice-Face Cross-modal Matching and Retrieval - A Benchmark, 2019
python works/11_SS_DIM_VFMR_Barlow.py --name=VFMR
-
Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association, CVPR, 2021
python works/5_Wen.py
-
Fusion and Orthogonal Projection for Improved Face-Voice Association, ICASSP, 2022
python works/6_FOP.py
-
Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast, IJCAI, 2022
python works/7_CMPC.py
-
Self-Lifting: A Novel Framework for Unsupervised Voice-Face Association Learning, ICMR, 2022
python works/9_SL.py
for self-lifting
python works/8_CAE.py
for the CCAE baseline
python works/11_SS_DIM_VFMR_Barlow.py --name=SL-Barlow
for the Barlow Twins baseline
Use wandb to view the training process:
-
Create a
.wb_config.json
file in the project root with the following content:{ "WB_KEY": "Your wandb auth key" }
-
Add
--dryrun=False
to the training command, for example:python main.py --dryrun=False