by Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Zhengdao Li, Long Hu, Kai Hwang.

This repository is for our Findings of NAACL 2022 paper 'Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation '. In this paper, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the instruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4.

• Dataset can be downloaded in Google Drive and processed version is here：Google Drive.
• Our trained model can be downloaded in Google Drive. If the trained model doesn't work or if there are any issues, please feel free to contact us.

pip install --editable .      
python prepare_data.py

nnictl create --config automl/config.yml -p 11111

cd postprocessing       
sh get_bleu4.sh

Please contact [email protected].