GPU Memory Benchmark about reformer-pytorch HOT 5 OPEN

Not sure how accurate these results are, but when I plot the memory usage with respect to the sequence length of a model with this setup dim = 128, depth = 6, heads = 4, batch_size = 4 I get this:

Note that for shorter sequences, the transformer model (use_full_attn flag to True) seems to be slightly less memory intensive, but starting with sequence length of 4096 the Reformer seems to work much better, while the transformer's memory soars and I run out of memory with a sequence length of 8192 (I have 11GB of memory), with the reformer I am able to get to up to a sequence length of 16384 without even filling the memory (Only uses 8GB) so we could probably get even higher.

from reformer-pytorch.

@pabloppp yup, the actual Transformer would have probably ceased to work at around 2048, because the reversibility is still in play even if you turn on full attention. other hyperparameters to play around with is ff_chunks, which you can increase for further CPU / memory trade offs

from reformer-pytorch.

I'm gonna add here the trials I do myself with the corresponding memory usage:

dim = 1024, seq_len=8960, depth=12, heads=16, batch_size=1: 8501MiB

from reformer-pytorch.

You should compare it to full attention! (Just set the use_full_attn flag to True)

from reformer-pytorch.

hi lucidrains.
How i pass parameter that says i want these many encoder and these many decoders.
Say 2 enc,2 decoder

from reformer-pytorch.