Enriching Language Models Representations via Knowledge Graphs Regularisation

Novel method for augmenting the representations learned by Transformer-based language models with the symbolic information contained into knowledge graphs. The model first compute the node embeddings of a knowledge graph via a deep graph network. Then it adds a new regularisation term to the loss of BERT that encourages the learned word embeddings to be similar to the node embeddings. The method method is tested on the challenging WordNet and Freebase knowledge graphs. The results show that the regularised embeddings perform better than standard embeddings on the chosen probing tasks.

Authors: Matteo Medioli

Credits: Deepak Nathani, Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

pip install -r requirements.txt

sh setup.sh

python kbgat/run_kbgat.py --get_2hop True --data kbgat/data/WN18RR

python main.py --epochs_gat 3000 --epochs_conv 200 --weight_decay_gat 0.00001 --get_2hop True --partial_2hop True --batch_size_gat 272115 --margin 1 --out_channels 50 --drop_conv 0.3 --weight_decay_conv 0.000001 --output_folder ./checkpoints/fb/out/ --data kbgat/data/FB15k-237

sh train_bert.sh