eliask93 / transformer-models-for-domain-specific-machine-translation Goto Github PK

Example application for the task of fine-tuning pretrained machine translation models on highly domain-specific translated sentences.

For this, likely translation pairs are first extracted from the original versions and the German book translations of the Harry Potter fantasy novel series using a Translated Sentence Mining approach. The extracted sentence translations are then used to fine-tune two baseline machine translation models (pre-trained model MarianMT for translation from English to German and Google's Text-To-Text Transfer Transformer T5).

Afterwards, some metrics are calculated to evaluate the performance boost from fine-tuning the models.

1. Split the unaligned txt files for each book and its translation file into sentences using Lingtrain Aligners splitter and preprocessor
2. Calculate language-independent sentence level embeddings for the split sentences using GoogleAI's Language-Agnostic BERT Sentence Embeddings (LaBSE) in the Sentence Transformers framework
3. Match the best fitting translation pairs for all sentences using K-Nearest Neighbors search, mostly following Sentence Transformers example application for Translated Sentence Mining
4. Filter the sentence pairs by a minimum similarity score
5. Remove sentence pairs containing sentences shorter than 20 or longer than 200 characters
6. Split the resulting corpus of ~54.000 likely parallel sentences randomly into a train, validation and test set (80%, 10%, 10%)

1. Load the pre-trained models Helsinki-NLP/opus-mt-en-de (MarianMTModel) and t5-base (T5ForConditionalGeneration) in huggingface
2. Fine-tune the models on the extracted parallel sentences using the train and evaluation set for 10 epochs each (training time: 03h-04m-45s for MarianMT and 09h-20m-10s for T5 on NVIDIA GeForce GTX 1660 Ti)

1. Use the non-fine-tuned MarianMT and T5 models to get machine translations for a sample from the test set
2. Use the fine-tuned models to get machine translations for a sample from the test set
3. Calculate BLEU, METEOR and BertScore between references and the target language translations for each the non-fine-tuned and the fine-tuned models

¹: setting the parameter rescale_with_baseline to True

• pytorch==1.7.1
• cudatoolkit=10.1
• pywin32

• transformers
• sentence_transformers
• faiss-gpu
• sacrebleu
• datasets
• bert-score
• lingtrain-aligner
• razdel
• dateparser
• python-dateutil
• numpy
• openpyxl

All files in this repository which contain text from the books are cut off after the first 50 rows. The trained model files pytorch_model.bin and optimizer.pt for each model are omitted in this repository.