A model for arabic-dialect-identification.

The source code for the paper: ABDELMAJEED Mohammed and Murtadha, Ahmed, et al. "A Three-Stage Neural Model for Arabic Dialect Identification" [1].

• The datasets used in our experminents can be downloaded from MADAR, QADI and NADI
• We have processed Nadi dataset datasets/Nadi/
• The embeddings of Nadi dataset for both cbow58 and arabvec download and put them to embedding/
• To build the embeddings and tokenizer of the other datasets:

  • download these models: * cbow58 * arabvec

  • unzip them into embedding/

Required packages are listed in the requirements.txt file:

pip install -r requirements.txt

• The data should be in the following format:

[
  {
      "text": "الفار العور يشوف فقط كيسي ومايشوف ماتويد",
      "label": "Iraq"
   }
]

• To train Bert-based ADI:
  • Run the following code

   python train.py --dataset Nadi --baseline camel --device_group 0
  

  • The params could be :

    • --baseline ={camel,arbert, mdb}
    • --dataset ={Nadi,QADI,Corpus-6, Corpus-9, Corpus-26}
    • --device_group ="0" # GPU ID

The the results will be written into results_bert.txt

• To train Word2vec-based ADI:

python train_LSTM.py --dataset Nadi --embeding cbow58 --device_group 0

• The params could be :

  • --embeding={cbow58, arabvec}
  • --dataset ={Nadi,QADI,Corpus-6, Corpus-9, Corpus-26}
  • --device_group ="0" # GPU ID

The the results will be written into results_lstm.txt

If you use the code, please cite the paper: