Speech Recognition with the caffe deep learning framework

UPDATE: We are migrating to tensorflow

This project is quite fresh and only the first of three milestones is accomplished: Even now it might be useful if you just want to train a handful of commands/options (1,2,3..yes/no/cancel/...)

1. training spoken numbers:

• get spectogram training images from http://pannous.net/spoken_numbers.tar (470 MB)
• start ./train.sh
• test with ipython notebook test-speech-recognition.ipynb or caffe test ... or <caffe-root>/python/classify.py
• 99% accuracy, nice!
• online recognition and learning with ./recognition-server.py and ./record.py scripts

Sample spectrogram, Karen uttering 'zero' with 160 words per minute.

2. training words:

• 4GB of training data
• net topology: work in progress ...
• todo: use upcoming new caffe LSTM layers etc
• UPDATE LSTMs get rolling, still not merged
• UPDATE since the caffe project leaders have a hindering merging policy and this pull request was shifted many times without ever being merged, we are migrating to tensorflow
• todo: add extra categories for a) silence b) common noises like typing, achoo c) ALL other noises

3. training speech:

• todo!
• 100GB of training data here: http://www.openslr.org/12/
• TIMIT dataset $27,000.00 membership fee or $250 for non-members+$2400 under research-only license?
• combine with google n-grams

Theoretical background: papers

A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In ICML, 2014

O. Vinyals, S. V. Ravuri, and D. Povey. Revisiting recurrent neural networks for robust ASR. In ICASSP, 2012

Andrew Ng et al / Baidu

Hinton et al / Toronto

good old Hinton

Schmidhuber et al using new 'ClockWork-RNNs'

The book: Automatic Speech Recognition: A Deep Learning Approach (Signals and Communication Technology) Hardcover – November 11, 2014 by Dong Yu (Author) and Li Deng (Author)

Related work

Also see the Kaldi project, which seems a bit messy but already uses deep learning with LSTM Another experimental LSTM network, which works out-of-the-box: Currennt