Comments (3)
- We just apply the selective SSM to each channel / hidden dimension separately, there's no concept of "head". In this sense it's similar to depthwise convolution. The in projection and output projections mixes the channels.
- There's no explicit position embedding. The model operates in a recurrent way:
hidden_state at time t+1 = function of hidden_state at time t and input at time t + 1.
In this sense it's similar to RNNs (e.g. LSTM). The model does have information on which input token comes first, which input token comes right after, etc.
Re: in-context learning: we actually don't know exactly the mechanism the model is using, all this stuff is new and there are lots to explore. One can hypothesize that as the model processes the prompt, it updates its hidden states to store important information, then use that to generate responses.
Transformer circuits describe 1 way transformers can do ICL. A very interesting question is whether other architectures also do ICL that way, or if there's another path.
from mamba.
Great, thank you Tri for the clarification. Very nice to not need positional embedding or individual heads.
from mamba.
Re: in-context learning: we actually don't know exactly the mechanism the model is using, all this stuff is new and there are lots to explore. One can hypothesize that as the model processes the prompt, it updates its hidden states to store important information, then use that to generate responses.
Transformer circuits describe 1 way transformers can do ICL. A very interesting question is whether other architectures also do ICL that way, or if there's another path.
My sense is in-context learning as a concept is at least aspirationally meant to refer to the ability of the model to complete a pattern A B .... A -> B, where A and B are not just tokens, but short token strings. In this setting, 'perfect generalization' would mean successfully predicting B when both A and B strings were never seen before in the training data. In such a setting one can argue from first principles there is no other way to solve this but the model learns to copy the information B from context to output.
In your paper the induction head task was of course much easier than this - a 16 token alphabet, and you are only trying to recall a single token, none of which are novel during testing. (Although the context as a whole is novel) It would be interesting to hit Mamba with the more challenging induction head task.
from mamba.
Related Issues (20)
- How to calculate Flops for mamba_chunk_scan_combined and mamba_split_conv1d_scan_combined function HOT 1
- Replace mamba1 with mamba2 and training becomes very slow! HOT 2
- Implement a "crossmamba" block for different length sequences HOT 3
- How to use sequence parallelism in Mamba2?
- Changing expansion_factor resulted in 'c10::DistBackendError HOT 1
- Mamba for encoder-decoder HOT 2
- mamba_split_conv1d_scan_combined error HOT 1
- torch.cuda.amp.custom_fwd(args...) Deprecated
- Is it possible to get ssm_states at specific seq len instead of just the last state? HOT 1
- How to modify mambaV2 to set causal=False? HOT 3
- Reducing _chunk_scan_bwd_kernel computation HOT 4
- Exploding gradients if ngroups is higher than 1. HOT 3
- clarification on how to interpret kernel size for conv1d HOT 1
- Some questions about the shape of A,B,C,D HOT 5
- How can I avoid using causal-conv1d? HOT 11
- Is it okay to put S6(MAMBA) and gated MLP blocks like a transformer? (Also please open the discussion tab)
- Vanishing gradient problem with more layer HOT 1
- Results vary greatly across experiments
- Gradient explosion in Mamba2 training, norm and loss divergence HOT 3
- Optimizing the bwd pass of Mamba 2 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mamba.