Training and inference code from the paper A Recurrent Neural Networks Approach for Estimating the Quality of Machine Translation Output

There are four subdirectories:

Using a parallel corpora, trains the neural machine translation (encoder-decoder-attention) model with two modifications 1) Both the encoder and decoder are bidirectional 2) introduce another hidden layer between maxout and the projection matrix in the decoder. During inference, instead of predicting the next word, we take the element-wise multiplication of the row vector from the projection matrix of the translated word and the additional hidden layer's node values from prediction to create an array of quality vectors (See paper for more details). Training data for quality estimation are then run through this model to obtain the quality vectors as input to qescore. This code was modified from the tensorflow seq2seq example.

The 2nd step to training the quality estimation model. Takes the quality vector from qualvec, runs a GRU RNN on it and use last hidden state to predict the HTER score. Currently uses L2 loss, which is unstable, but should modify to use loss function from this paper for better stability.

Similar to qualvec except uses "general" attention from Effective Approaches to Attention-based Neural Machine Translation which worked better. Will merge with qualvec.

Get the baseline score using Support Vector Regression on baseline features. Hyperparameters are optimized using particle swarm optimization.

To Train:

Tokenize text with the moses tokenizer

For training NMT model:

python qualvec/qualvec.py --data_dir <data directory containing parallel corpora> --train_dir <training dir>

For obtaining quality vectors

python qualvec/qualvec.py --data_dir <data directory containing quality estimation data> --train_dir <training dir> --qualvec

training quality estimation model

python qescore/qe.py --data_dir <data directory containing quality vectors and labels> --train_dir <training dir>

quality estimation inference

python qescore/qe.py --data_dir <data directory containing training vectors> --train_dir <training dir> --qescore

See source code for additional options