Comments (5)
Not sure how accurate these results are, but when I plot the memory usage with respect to the sequence length of a model with this setup dim = 128, depth = 6, heads = 4, batch_size = 4
I get this:
Note that for shorter sequences, the transformer model (use_full_attn
flag to True
) seems to be slightly less memory intensive, but starting with sequence length of 4096 the Reformer seems to work much better, while the transformer's memory soars and I run out of memory with a sequence length of 8192 (I have 11GB of memory), with the reformer I am able to get to up to a sequence length of 16384 without even filling the memory (Only uses 8GB) so we could probably get even higher.
from reformer-pytorch.
@pabloppp yup, the actual Transformer would have probably ceased to work at around 2048, because the reversibility is still in play even if you turn on full attention. other hyperparameters to play around with is ff_chunks
, which you can increase for further CPU / memory trade offs
from reformer-pytorch.
I'm gonna add here the trials I do myself with the corresponding memory usage:
dim = 1024, seq_len=8960, depth=12, heads=16, batch_size=1: 8501MiB
from reformer-pytorch.
You should compare it to full attention! (Just set the use_full_attn
flag to True
)
from reformer-pytorch.
hi lucidrains.
How i pass parameter that says i want these many encoder and these many decoders.
Say 2 enc,2 decoder
from reformer-pytorch.
Related Issues (20)
- TOKEN_SELF_ATTN_VALUE and QK attention
- Music Reformer results (some problems) HOT 1
- The exact English pretraining data and Chinese pretraining data that are exact same to the BERT paper's pretraining data.
- Image as input
- why lucidrains use reformerlm as pre train model?
- Class token implementation HOT 1
- Error while testing Code in Readme HOT 2
- Is the argument max_seq_len in Reformer() used? HOT 1
- Reversible layers increase memory usage HOT 1
- self-supervised.py Training Script is it worthwhile pretokenizing ?
- fp16 support
- About sequence length
- Default embeddings are rotary Embeddings not Axialpositional Embeddings as stated by the documentation HOT 1
- error in eval loss & ppl
- half precision on long sequence doesn't work HOT 3
- FLOPs calculation for LSHSelfAttention in LSH mode and Full attention mode
- not find axial_positional_embedding.py HOT 2
- ImportError: cannot import name 'amp' from 'apex' HOT 1
- how to get the dataset of enwik8.gz ?
- Is the unsort logits necessary?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from reformer-pytorch.