A model for arabic-dialect-identification.
The source code for the paper: ABDELMAJEED Mohammed and Murtadha, Ahmed, et al. "A Three-Stage Neural Model for Arabic Dialect Identification" [1].
- The datasets used in our experminents can be downloaded from MADAR, QADI and NADI
- We have processed Nadi dataset datasets/Nadi/
- The embeddings of Nadi dataset for both cbow58 and arabvec download and put them to embedding/
- To build the embeddings and tokenizer of the other datasets:
Required packages are listed in the requirements.txt file:
pip install -r requirements.txt
- The data should be in the following format:
[
{
"text": "الفار العور يشوف فقط كيسي ومايشوف ماتويد",
"label": "Iraq"
}
]
- To train Bert-based ADI:
- Run the following code
python train.py --dataset Nadi --baseline camel --device_group 0
-
The params could be :
- --baseline ={camel,arbert, mdb}
- --dataset ={Nadi,QADI,Corpus-6, Corpus-9, Corpus-26}
- --device_group ="0" # GPU ID
The the results will be written into results_bert.txt
- To train Word2vec-based ADI:
python train_LSTM.py --dataset Nadi --embeding cbow58 --device_group 0
-
The params could be :
- --embeding={cbow58, arabvec}
- --dataset ={Nadi,QADI,Corpus-6, Corpus-9, Corpus-26}
- --device_group ="0" # GPU ID
The the results will be written into results_lstm.txt
If you use the code, please cite the paper: