📜 Paper | 📝 Blogpost | 🤗 HF Model 🤗 HF Dataset | 🤗 HF Benchmarks

We release Airavata v0.1, a Hindi chat model instruction finetuned on SarvamAI's OpenHathi. Please refer to our official blogspot for our model details, dataset creation and evaluation process.

Cover image is generated by DALL-E 3

This repo was forked from allenai/open-instruct, an open-source initiative for instruction-tuning widely used pretrained language models. More instructions about the codebase can be found there.

To run training, evaluation, or inference for our finetuned models, you need to install the required packages by running the following command (after installing pytorch):

pip install -r requirements.txt

If you just want the dependencies for the weight diff script, use:

pip install -r weight-diff-requirements.txt

In general, any model from the Hugging Face Hub may be loaded for training. You may train a model from scratch or finetune an existing model. The following code snippet shows our command for training SarvamAI's OpenHathi base model.

# install additional package
pip install lm-dataformat

# preprocess and tokenize the datasets before you perform training from scratch
python3 scripts/tokenize_dataset.py \
    --tokenizer_path <tokenizer_path> \
    --data_path <data_path> \
    --save_path <save_path> \
    --max_seq_length <max_seq_length> \
    --max_examples <max_examples> --max_tokens <max_tokens>

bash scripts/train_with_accelerate.sh

bash scripts/finetune_lora_with_accelerate.sh

Please check the scripts for more information on the arguments.

We cover various instruction datasets to train our chat model. The collection consists of:

• Anudesh
• wikiHow
• Flan v2 (67k sample subset)
• Dolly
• Anthropic-HHH (5k sample subset)
• OpenAssistant v1
• LymSys-Chat (50k sample subset)

We have put together the above datasets and it can be accessed from Hugging Face

python3 reformat_indic_instruct_data.py --output_dir <output_dir>

We have evaluated on standard Indic and English benchmarks to assess the capabilities of our model. The benchmarks are:

• Indic NLU and Commonsense Reasoning tasks
  • IndicSentiment
  • IndicCOPA
  • IndicXNLI
  • IndicXParaphrase
• Indic NLG
  • IndicQA
  • IndicHeadlineGeneration
  • IndicWikiBio
• English NLU and Commonsense Reasoning tasks
  • MMLU
  • BoolQ
  • ARC Easy (Both Easy and Challenge subsets)
  • Hellaswag
• English-Hindi Translation
  • Flores
  • IN22-Gen

In addition, we also evaluate on the Indic benchmarks using the translate-test approach ie. evaluating the Hindi language benchmarks by translating it to English on an English model. For this, we use the LLaMA-2 7B chat model. Note that OpenHathi base itself was finetuned on LLaMA-2 7B base, hence it was appropriate to compare our model against its English counterpart.

Similarly, we translate the English benchmarks to Hindi using IndicTrans2 and evaluate them on our model. Note that both OpenHathi and Airavata were trained bilingually on English and Hindi, so they support generation in both languages.

You would have to request access to use the LLaMA variants and log in to the huggingface hub (or pass a token). This process is detailed in the Hugging Face documentation.

Most of the aforementioned datasets can be directly loaded from Huggingface Hub. However, please the extract the MMLU (English and Hindi Translated) variant provided in the data/eval directory for evaluations.

The evaluation scripts for the benchmarks listed can be found at scripts/indic_eval/name_of_the_task.sh for Indic benchmarks and scripts/translate_test_eval/name_of_the_task.sh for translate-test. The following command shows how you can evaluate on IndicSentiment

# Evaluation on IndicSentiment (Hindi) on a 5-shot setting
python3 -m eval.indicsentiment.run_eval \
    --ntrain 5 \
    --save_dir "results/indicsentiment/airavata-5shot" \
    --model_name_or_path ai4bharat/airavata \
    --tokenizer_name_or_path ai4bharat/airavata \
    --eval_batch_size 4

# Evaluation on IndicSentiment (Translate-test) on a 5-shot setting
python3 -m eval.indicsentiment.run_translate_test_eval \
    --ntrain 5 \
    --save_dir "results/translate_test/indicsentiment/llama2-chat-5shot" \
    --model_name_or_path meta-llama/Llama-2-7b-chat-hf \
    --tokenizer_name_or_path meta-llama/Llama-2-7b-chat-hf \
    --eval_batch_size 4

Our chat model is made available on Hugging Face. The model is licensed under the original Llama license.

As for the amount of parameters, LLaMA-2 7B was used to train the OpenHathi (and subsequently Airavata), hence it comprises of 7 billion parameters in total.

In case of any queries or issues, we recommend you to open issues on the GitHub repo directly.

If you used this repository or our models, please cite our work:

@article{gala2024airavata,
  title   = {Airavata: Introducing Hindi Instruction-tuned LLM},
  author  = {Jay Gala and Thanmay Jayakumar and Jaavid Aktar Husain and Aswanth Kumar M and Mohammed Safi Ur Rahman Khan and Diptesh Kanojia and Ratish Puduppully and Mitesh M. Khapra and Raj Dabre and Rudra Murthy and Anoop Kunchukuttan},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2401.15006}
}