Comments (4)
DanFu09
commented on August 20, 2024
This is something we're very interested in and still working on! We don't have a formula for it quite yet.
from m2.
autumn-2-net
commented on August 20, 2024
This is something we're very interested in and still working on! We don't have a formula for it quite yet.
This doesn't sound like good news, it looks like I'll just have to CROSS ATTENTION mix MonarchMixer, is there a performance loss compared to raw ATTENTION?
from m2.
DanFu09
commented on August 20, 2024
We've seen that we can match self-attention in quality with some gated convolutions (see the paper for details). Cross attention is still an open problem - which we'll be working on!
from m2.
autumn-2-net
commented on August 20, 2024
We've seen that we can match self-attention in quality with some gated convolutions (see the paper for details). Cross attention is still an open problem - which we'll be working on!
If I use M2 can I not use positional coding as I feel that M2 looks a bit similar to conv which allows the model to know the positional information
from m2.
Related Issues (19)
- `python3 test_flash_mm.py` got error HOT 5
- Is there any detailed explanation of M2-VIT?
- What should I do if the √N is not an integer? HOT 3
- M2 model is applied on single image deraining model based on transformer HOT 5
- MNLI yaml config
- Missing Licence HOT 1
- does M2 work with ONNX? HOT 3
- Why is there such a big difference in cosine similarity between embeddings of the same pair when using padding=max_length versus padding=true? HOT 2
- Multilingual?
- precision on imagenet experiment HOT 1
- A question on square matrices
- LoCo Benchmark - BM25 & Insights HOT 5
- Code for projecting pre-trained BERT weights into Monarch matrices HOT 2
- Bert-like implementation HOT 4
- Some minor inconsistency in the paper HOT 2
- Code for model architecture
- Will M2-GPT be open-sourced? HOT 6
- MonarchMixerLayer HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from m2.