This repo contains the code for two papers:

• Feedback Transformer
• Expire-Span

The training code is structured for long sequential modeling with Transformer-like architectures.

You will need a CUDA-enabled GPU to run the code.

Run the following:

pip install -r requirements.txt

Introduced in Addressing Some Limitations of Transformers with Feedback Memory.

Numbers are Bits-Per-Character

bash experiments/feedback/enwik8.sh

Numbers are % Accuracy on Test

bash experiments/feedback/algorithmic_3var.sh
bash experiments/feedback/algorithmic_5var.sh

Introduced in Not All Memories are Created Equal: Learning to Expire.

Numbers are Bits-Per-Character

bash experiments/expire_span/enwik8.sh

bash experiments/expire_span/object_collision_16k.sh
bash experiments/expire_span/object_collision_32k.sh
bash experiments/expire_span/object_collision_64k.sh

The code is licensed under CC-BY-NC license. See the LICENSE file for more details.