Code for our paper Kformer: Knowlede Injection in Transformer Feed-Forward Layers

The project is based on Fairseq.

To install requirements:

cd fairseq
./setup.sh

mkdir models
cd models
wget https://dl.fbaipublicfiles.com/fairseq/models/roberta.base.tar.gz
tar -zxvf roberta.base.tar.gz

Use the command below to finetune SocialIQA on Kformer. You can change the layer to inject by editing the arg --knowledge_layer. --knowledge_layer contains two arguments [a,b) denoting the interval of the layer of Roberta.

./fairseq/run_social.sh

Use the command below to finetune MedQA on Kformer.

./fairseq/run_med.sh

Use the following command to evalute the finetuned model. Set the --knowledge_layer the same as the arg during finetuning.

export ModelPath = $ModelPath$
export DataPath = $DataPath$
python fairseq/test_social.py --model_path $ModelPath$ --knowledge_layer 9 12 --data_file $DataPath$

Change fairseq/test_social.py to test_med.py to evaluate MedQA.

Please give us a ⭐ and cite our paper as

@article{Yao2022KformerKI,
  title={Kformer: Knowledge Injection in Transformer Feed-Forward Layers},
  author={Yunzhi Yao and Shaohan Huang and Ningyu Zhang and Li Dong and Furu Wei and Huajun Chen},
  journal={ArXiv},
  year={2022},
  volume={abs/2201.05742}
}