devsinghsachan / multilingual_nmt Goto Github PK

This codebase was used for the multilingual translation experiments for the paper "Parameter Sharing Methods for Multilingual Self-Attentional Translation Models, WMT-EMNLP 2018".

The multilingual model is based on the Transformer model and also contains the following features:

• positional encoding
• multi-head dot-product attention
• label smoothing
• warm-up steps based training of Adam optimizer
• shared weights of the embedding and softmax layers
• beam search with length normalization
• exponential moving average checkpoint of parameters

One can install the required packages from the requirements file.

pip install -r requirements.txt

• Download the TED talks dataset as:

bash download_teddata.sh

This command will download, decompress, and will save the train, dev, and test splits of the TED talks under data directory.

• One can use the script ted_reader.py to specify language pairs for both bilingual/multilingual translation tasks.

• For bilingual/multilingual translation, just specify the source and target languages as

python ted_reader.py -s ja en zh fr ro -t en zh fr ro ja

• For multilingual translation, by default the training data will consist of the cartesian product of all the source and target language pairs.
• If all possible combinations of the language pairs are not needed, then just use the option of -ncp

python ted_reader.py -s ja en zh fr ro -t en zh fr ro ja -ncp

• Above command will only create training data for the corresponding language pairs, i.e. [(ja, en), (en, zh), (zh, fr), (fr, ro), (fr, ja)]

• For evaluating the multiingual model, one can generate the test set for each bilingual pair using the above command.

For convenience, there are some example shell scripts under tools directory

• Bilingual Translation (NS)

bash tools/bpe_pipeline_bilingual.sh src_lang tgt_lang

• Fully Shared Multilingual Translation (FS)

bash tools/bpe_pipeline_fully_shared_multilingual.sh src_lang tgt_lang1 tgt_lang2 

• Partial Sharing Multilingual Translation (PS)

bash tools/bpe_pipeline_MT.sh src_lang tgt_lang1 tgt_lang2 share_sublayer share_attn

An example of sharing the Key(k), Query(q) in both the attention layers (Self, Source)

bash tools/bpe_pipeline_MT.sh src_lang tgt_lang1 tgt_lang2 k,q self,source

• Dataset Statistics

• Bilingual Translation Tasks

• Multilingual Translation Tasks

If you find this code useful, please consider citing our paper as:

@InProceedings{devendra2018multilingual,
  author = 	"Sachan, Devendra
		and Neubig, Graham,
  title = 	"Parameter Sharing Methods for Multilingual Self-Attentional Translation Models",
  booktitle = 	"Proceedings of the Third Conference on Machine Translation",
  year = 	"2018",
  publisher = 	"Association for Computational Linguistics",
  location = 	"Brussels, Belgium"
}