The model and theory behind our implementations in this library are explained in our ICML paper "Improving Transformers with Probabilistic Attention Keys". A short summary of our paper is given on our blog.
Instructions for running the experiments in our paper are given in the following files:
lra/README.md
language-modeling/lmtool-fwms/README.md
Please cite us at
@inproceedings{nguyen2022improving,
title={Improving Transformers with Probabilistic Attention Keys},
author={Nguyen, Tam and Nguyen, T and Le, D and Nguyen, K and Tran, A and Baraniuk, R and Ho, Nhat and Osher, S},
year={2022},
booktitle={Proceedings of the 39th International Conference on Machine Learning},
series={Proceedings of Machine Learning Research}
}