These are different files I created to do different tasks when I was working on creating ASR model for mTEDx dataset. The following is a description of the different files you can find in this repo:
download_mted.sh: A bash script to download the ASR part of the mTEDx dataset.process_mtedx: A file to process the mTEDx dataset. By processing, I mean:- Split the audio of the ted talks found in the datasets into shorter segments that can be used for training ASR models.
- Normalizes the audio files to a sample rate of
16000and number of channels of1. - Align between the audio segments and the text.
stats.xlsx: An excel file gathering all the details of the data found in the dataset.
You can download mTEDx data by running the following bash script:
bash download_mtedx.shAfter downloading the audio data, you need to process the audio data and split the whole talks into different segments and align it with the correct text. All of that can be done by running this python script like so:
python process_mtedx.py \
--in [IN_DATA_PATH] \
--out [OUT_DATA_PATH] \
--langs [LANGS] \
--groups [GROUPS]
For example:
python process_mtedx.py \
--in /scratch/1/user/manwar/data/mTEDx \
--out /scratch/1/user/manwar/data/mTEDx_wav \
--langs ar el es de fr it pt ru\
--groups test valid train
This will process all audio data and organize it in the following tree:
mTEDx
├── ar
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
├── de
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
├── el
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
├── es
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
├── fr
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
├── it
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
├── pt
│ ├── test
│ ├── test.json
│ ├── train
│ ├── train.json
│ ├── valid
│ └── valid.json
└── ru
├── test
├── test.json
├── train
├── train.json
├── valid
└── valid.json
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent