CMixAugment-MT: An Empirical Study of Leveraging Finetuning of Machine Translation Model To Produce Synthetic Code Mixing Data For Data Augmentation
This is my NLP804 (Deep Learning For Natural Language Generation) project. The goal here is to evaluate different finetuning strategies on machine translation model for producing better code-mixed synthetic dataset. Due to availability on the parallel corpus, I only conducted the studies for Hinglish (Hindi-English) translation from English.
Setup conda environment
conda create -n cmixaugment-mt python=3.10
conda activate cmixaugment-mt
Install all dependencies
pip install -r requirements.txt
This project mainly uses hydra
library to store the configuration. To run the experiment you can run this command
python main.py --config-name <adapter_config/baseline_config/sft_config>
We use various evaluation metrics to assess the quality of the generated synthetic data. For downstream task evaluation, you can check the eval/GLUECoS
on NLI
and Sentiment Analysis
(note that training file other than baseline, we add sample ratio information before the extension in the filename). While for the code-mixing metrics, you can check the eval/LID-tool
and the main file is eval/LID-tool/measure_cmi.py
. For evaluation using LLM-as-a-judge, you can check eval/LLM-eval
and we use llm_eval.py
to evaluate the text quality using specific LLM and then merge all the LLMs' judgment using merge_llm_scores.py
. Lastly, for reference-free metrics you can check both eval/eval_comet_score.py
and eval/eval_lbse_sim_score.py
.
You can check out this link