May I know how to do sentence-level lip reading? about lipreading_using_temporal_convolutional_networks HOT 3 CLOSED

Hi, sentence-level speech recognition is quiet different from word-level speech recognition. To be honest, I don't suggest you do it by segmenting sentences since a sequence may have different labels. Besides, word based classification has an issue of out of vocabulary words.

from lipreading_using_temporal_convolutional_networks.

@mpc001 Thank you very much for your reply. I have another two questions:

1. Do you have any suggestions on sentence-level lipreading?
2. What's the running time of the models listed in your Model Zoo? I tested the resnet18_mstcn_adamw_s3, snv1x_dsmstcn3x and snv1x_tcn1x models. The running time of three models are 10ms, 20ms and 10 ms on my machine. Did you have a similar trend? I thought snv1x_tcn1x has the least flops, so it should be the fastest model.

Thank you very much!

from lipreading_using_temporal_convolutional_networks.

Extending this to sentence-level lipreading is possible but not straightforward. You can think of what we've done here as getting a strong representation, but then you need to add something on top that instead of predicting single labels per segment predicts continuous speech. Maybe you should look into works that have experiments on LRS2 for example to learn how to do that. The head of the architecture would change and the loss is different. If you want to get the best out of it, you can apply a language model.

from lipreading_using_temporal_convolutional_networks.