Code base for paper Hierarchical pointer-generator on Confusion Network: Natural AnswerGeneration from Speech.
All dependencies can be installed via:
pip install -r requirements.txt
Pad the answer text file.
python add_padding.py --input <answer-filename> --output <padded-answer-filename> --padding <num of pad>
onmt_preprocess -train_ques data/ques-train.txt -train_ans data/ans-train.txt -train_tgt data/tgt-train.txt -valid_ques data/ques-val.txt -valid_ans data/ans-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo --dynamic_dict --share_vocab
The data consists of parallel source (src
) and target (tgt
) data containing one sentence per line with tokens separated by a space:
ques-train.txt
: text file containing path to a question confusion network in each lineans-train.txt
: text file containing the padded factoid answer in each linetgt-train.txt
: text file containing the target full length answerques-val.txt
: text file containing the path to a validation question confusion network in each lineans-val.txt
: validation text file containing the path to the padded factoid answer in each linetgt-val.txt
: validation text file containing the target full length answer
Validation files are required and used to evaluate the convergence of the training. It usually contains no more than 5000 sentences.
After running the preprocessing, the following files are generated:
demo.train.pt
: serialized PyTorch file containing training datademo.valid.pt
: serialized PyTorch file containing validation datademo.vocab.pt
: serialized PyTorch file containing vocabulary data
Internally the system never touches the words themselves, but uses these indices.
onmt_train -data data/demo -save_model demo-model -word_vec_size 300 -model_type lattice -encoder_type brnn -layers 2 -rnn_size 512 \
-data data/demo -batch_size 32 -valid_batch_size 32 -valid_steps 2500 -dropout 0.5 -start_decay_steps 10000 -coverage_attn -copy_attn \
--share_embeddings
onmt_translate -model demo-model_acc_XX.XX_ppl_XXX.XX_eX.pt -ques data/ques-test.txt -ans data/ans-test.txt -output pred.txt -replace_unk -verbose -beam 5
The codebase is built over OpenNMT. OpenNMT-py is run as a collaborative open-source project.