depth_growing_nmt's Introduction

Depth Growing for Neural Machine Translation

This repository is the code for short paper of ACL 2019: Depth Growing for Neural Machine Translation.

The project is based on the fairseq (version 0.5.0). (Please get familar with the fairseq project first)

@inproceedings{wu2019depth,
  title={Depth Growing for Neural Machine Translation},
  author={Wu, Lijun and Wang, Yiren and Xia, Yingce and Tian, Fei and Gao, Fei and Qin, Tao and Lai, Jianhuang and Liu, Tie-Yan},
  booktitle={ACL 2019},
  year={2019}
}

Requirements and Installation

A PyTorch installation (0.4.0) and install fairseq with:

pip install -r ./deepNMT/requirements.txt
python ./deepNMT/setup.py build develop

Data

Please refer to WMT14_En_De for data processing.

Training

The detaied training procedure is:

Train shallow model with six layers

train_fairseq_en2de.sh

Train first several steps of the deep model with eight layers. For example, train only 10 steps.

train_fairseq_en2de_deep.sh

Prepare the deep model. Initialize the deep model with the parameters from the shallow model in last step.

build_initial_ckpt_for_deep.sh

Reload the initialized deep model and train deep model with eight layers.

train_fairseq_en2de_deep.sh

Inference

The detailed inference procedure is:

bash infer_deepNMT.sh 0 <shallow_model_ckpt_path>  <deep_model_ckpt_path>

depth_growing_nmt's People

Contributors

Stargazers

Watchers

depth_growing_nmt's Issues

Wondering about the freeze operation

Hi,
I'm really interested in your work and fascinated by the strong improvement in your paper. But when reading your code, I have the following question:

If I understand the scripts correctly, you first train the shallow model, then replace the parameter in the deep model. Finally, you directly train the deep model. However, I didn't see any code about the operation about freeze operation as you described in your paper.

"in Stage 1, the bottom module (i.e., enc1 and dec1) is trained and subsequently holds constant; in Stage 2, only the top module (i.e., enc2 and dec2) is optimized."

Am I missing some details in your code?

Looking forward to your reply. Thanks

apeterswu / depth_growing_nmt Goto Github PK

depth_growing_nmt's Introduction

Depth Growing for Neural Machine Translation

Requirements and Installation

Data

Training

Inference

depth_growing_nmt's People

Contributors

Stargazers

Watchers

Forkers

depth_growing_nmt's Issues

Wondering about the freeze operation

train first several steps

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent