Comments (4)
lucidrains
commented on May 26, 2024
just need the init_deepnet(retro) function now 1c57013
from retro-pytorch.
lucidrains
commented on May 26, 2024
made the residual scaling customizable because with the cross attention in RETRO being every 3rd layer, it may end up being different than what is in the paper
from retro-pytorch.
lucidrains
commented on May 26, 2024
done
from retro-pytorch.
OhadRubin
commented on May 26, 2024
Hey @lucidrains,
I'm trying to replicate RETRO in Jax, and it isn't going well.
How crucial was the deepnet init to convergence?
Because right now, my model can't even solve a very simple task.
(The task is reversing a string that is encrypted using a simple cypher, where the cypher is given as the neighbor tokens, so a model without cca can't solve it).
from retro-pytorch.
Related Issues (20)
- Error Reconstructing FAISS Index HOT 18
- Unable to do non-retrieval sampling
- Extra layer encoder_output_to_decoder_dim cause issue with distributed training HOT 2
- TrainingWrapper does not support line breaks HOT 8
- RuntimeError: Error in void faiss::gpu::GpuIndexIVFPQ::verifySettings_() HOT 3
- Double [CLS] token in the first doc chunk HOT 1
- Retro-fitting a pretrained model HOT 7
- Clarification on Architecture
- Scann vs faiss HOT 6
- 'NoneType' object is not callable HOT 1
- Is there any pre-trained RETRO model released yet? HOT 4
- Huggingface model
- I am revising the model to solve QA task.. HOT 1
- How to give Prompt to trained RETRO Model? HOT 6
- Why are there so many position embeddings? HOT 5
- Causal mask in Chunked Cross Attention
- Error # could not open .tmp/.index/knn.index for reading: No such file or directory
- Question-Answer Dataset Format ?
- AttributeError: module 'faiss' has no attribute 'GpuParameterSpace' HOT 2
- Question: residual connect after `ChunkedCrossAttention`? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from retro-pytorch.