mtanghu / leap Goto Github PK
View Code? Open in Web Editor NEWLEAP: Linear Explainable Attention in Parallel for causal language modeling with O(1) path length, and O(1) inference
License: Creative Commons Zero v1.0 Universal
LEAP: Linear Explainable Attention in Parallel for causal language modeling with O(1) path length, and O(1) inference
License: Creative Commons Zero v1.0 Universal
Transformers are RNNs (link: https://arxiv.org/pdf/2006.16236.pdf) talks about constant memory gradient computations which should be entirely realizeable for this project (the math is reasonably similar in structure at least).
Seemingly currently the memory usage does scale with sequence length though this may just be because larger inputs will need more memory to store all the embeddings. To that extent, it shouldn't help the training that's done in parallel.
This will be important for infinite context in the RNN formulation though! (see #14)
Rotary embeddings, key query embeddings, or maybe an adapted ALiBi embedding all seem worth implementing to improve the performance of LEAP. Ideally this would be implemented as an option that can be passed in the config like BERT's position_embedding_type
(link below) maybe even the same code could be used!
This should also enable Infinite Context Length, though constant memory gradient calculation (see #12) would be needed to really allow for Infinite Context Length.
https://huggingface.co/docs/transformers/v4.21.1/en/model_doc/bert#transformers.BertConfig
It seems no matter how many attempts are done to fix the equation rendering, they seem to still not work on all browsers. Currently the equations seems to render about right for Firefox, but not Chrome.
Possible path forward:
Would be good to experiment with rescaling attention values, say with full attention as well to see if you can prove something about how the attention values may be what causes training instability (the current theory is that large language models can easily be pushed into local minima that are hard to break out of when the attention values get too big before being softmax-ed). Like maybe the experiment would be, use large dimensions, and to save on memory, only use a single head and only a single layer of multihead attention (with feed forward layers after it) with comically small sequence lengths (of like 4) and small batch sizes (also use no dropout to stop regularization). Consider also only using a subset of wikitext-2. Show how larger dimensions make effect worse. Measure the pre-softmax values and see if they get larger over time.
Would also be good to see if you can try a different implementation of rescaling for LEAP could work better since it's still oddly unstable for how strongly scaled it should be.
The current querying system of LEAP seems quite inflexible. Like take for example the winograd task of knowing what a pronoun refers to. For the pronoun token information to make it to the prediction token, the query vector for that prediction token would need to match the focus vector, thus requiring the prediction to already "know" the pronoun apriori.
Using a sigmoid gate would make sense (to replace the querying), though I'm concerned about explainability, and if the sigmoid will saturate at large scales.
A different tempting option would be to create focus weighted keys and focus weighted values which will be queried. This would require a total of 5 linear projections though...
I think one of the Fs can be shared though?, i.e. LEAP = Q \cdot w-Focus(F, K, K) * w-Focus(F, K, V) -- is this too complicated though?
Title. It's more standard to have use bibtex citations especially on github and would improve crediting of other work
It is suboptimal to have to update the README often to reflect development changes. It would be better to move these to issues or projects, and instead create a "possibilities" section which still notes the different possibilities for development.
Would help to have contribution guidelines
Still working on the follow change to finish
The README currently claims that it keeps attention sparse and improves training stability (there is already a config option to turn it on and off/play with the rescaling constant). It would be good to show this data, as well as try implementing it with normal transformers (say GPT2) where I think it should actually mildly improve scaling performance as the dot products won't get pushed into low gradient areas of the softmax.
I think these experiments would go well in a folder named "Experiments" in the repo or something along those lines.
even when i adjust the scaling in the FastAttention class the loss is still become NaN after training for few hours
Some original ideas will likely be needed, but in general this should be achievable simply through measuring the dot-product similarities of F1 and F2 (and also softmaxing them) to see what tokens the model is paying attention to, then factor in the querying aspect.
A good entry point would be to reproduce a variation of the pairwise Attention diagram found in the appendix of Attention is All you Need
In general the decoder formulation (which is already done) should be harder than the encoder formulation. One can always just perform LEAP in both directions (like a bidirectional RNN), though I'm curious if there is a more efficient implementation.
An Attention mechanism is considered Turing Complete if it can model any kind of sequence computation. Global LEAP with
If this can be shown, it may be of less importance to perform one-to-one comparisions with GPT2 as there is theory to back up the expressiveness of the architecture/attention mechanism
Some inline equations do render, some don't. I've found that two inline equations in the same sentence do NOT render either. I've found also that potentially delimiting with bullet points can occasionally help. Still can't seem to crack it though
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.