lip2speech's Introduction

Lip2Speech [PDF]

A pipeline for lip reading a silent speaking face in a video and generate speech for the lip-read content, i.e Lip to Speech Synthesis.

Video Input	Processed Input	Speech Output

Architecture Overview

LRW

Alignment Plot	Melspectogram Output

Usage

Demo

The pretrained model is available here [265.12 MB]

Download the pretrained model and place it inside savedmodels directory. To visulaize the results, we run demo.py.

python3 demo.py

Default arguments

dataset: LRW (10 Samples)
root: Datasets/SAMPLE_LRW
model_path: savedmodels/lip2speech_final.pth
encoding: voice

Evaluate

Evaluates the ESTOI score for the given Lip2Speech model. (Higer is better)

python3 evaluate.py --dataset LRW --root Datasets/LRW --model_path savedmodels/lip2speech_final.pth

Train

To train the model, we run train.py

python3 train.py --dataset LRW --root Datasets/LRW --finetune_model_path savedmodels/lip2speech_final.pth

finetune_model_path - Use as base model to finetune to dataset. (optional)

Acknowledgement

tacotron2

Citation

If you use this software in your work, please cite it using the following metadata.

@software{Millerdurai_Lip2Speech_2021,
author = {Millerdurai, Christen and Abdel Khaliq, Lotfy and Ulrich, Timon},
month = {8},
title = {{Lip2Speech}},
url = {https://github.com/Chris10M/Lip2Speech},
version = {1.0.0},
year = {2021}
}

lip2speech's People

Contributors

Stargazers

Watchers

lip2speech's Issues

How to train the model on Custom Dataset

Hello, I hope you are doing well. I want your help in training the model on our custom dataset, where we have taken videos from 15 speakers, speaking in Urdu. Its one second long video, where speaker is just speaking one word. In total I have 2700 videos

MelSpec2Audio not found anywhere on the internet

Hey Chris10M,
I can't seem to find MelSpec2Audio anywhere on the internet except for its single occurrence in evaluate.py file. Can you please guide me through this error?
Thank you

p.s.
error:
2023-03-21 11:20:10 | INFO | numexpr.utils | NumExpr defaulting to 2 threads.
Traceback (most recent call last):
File "/content/drive/MyDrive/Colab Notebooks/lip2speech/Lip2Speech-main/demo.py", line 9, in
from datasets import MelSpec2Audio
ImportError: cannot import name 'MelSpec2Audio' from 'datasets' (/usr/local/lib/python3.9/dist-packages/datasets/init.py)

Video Preprocessing

Hi,

Nice work!

Could you point me towards the code you use to crop the face and mouth from the LRW samples into the .npz files found at Datasets/SAMPLE_LRW ? I'd like to try it out with my own data

Thanks

Recommend Projects

chris10m / lip2speech Goto Github PK