Comments (13)
lucidrains
commented on May 23, 2024
1
ok, i'm closing this, feel free to reach out to Erik or Kip (at Eleuther) if you are interested in contributing towards an open sourced model
from retro-pytorch.
lucidrains
commented on May 23, 2024
yup, @enijkamp is planning on doing so, but he will first test this repo and then port it over to jax
potentially 7B parameters i'm told, and he is going to push for open source 🥳
from retro-pytorch.
malteos
commented on May 23, 2024
Glad to hear that. I'd be happy to contribute. Are there any particular issues that need some help?
from retro-pytorch.
lucidrains
commented on May 23, 2024
@malteos best to reach out to Erik, as he is doing the training ;)
from retro-pytorch.
paraschopra
commented on May 23, 2024
maybe you can collaborate with elutherai. having a large (7b) open source
model with a huge index powered by retro will be awesome.
Regards,
Paras Chopra
@paraschopra <https://twitter.com/paraschopra> on twitter
…On Fri, 11 Feb 2022 at 05:49, Phil Wang ***@***.***> wrote:
@malteos <https://github.com/malteos> best to reach out to Erik, as he is
doing the training ;)
—
Reply to this email directly, view it on GitHub
<#4 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAF2BKJGM63CTTWB42555LTU2RIXZANCNFSM5OBYI4NQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
from retro-pytorch.
lucidrains
commented on May 23, 2024
@paraschopra Yup, someone at Eleuther is eyeing the paper (and probably going to use my repo) - so if Erik doesn't fall through, there's them
from retro-pytorch.
ronald-d-rogers
commented on May 23, 2024
Do you all plan on open sourcing the world knowledge somehow as well?
from retro-pytorch.
lucidrains
commented on May 23, 2024
@ronald-d-rogers do you mean the retrieval database?
from retro-pytorch.
ronald-d-rogers
commented on May 23, 2024
Yes, perhaps one for this new Pile?
…On Sat, Feb 12, 2022, 11:00 PM Phil Wang ***@***.***> wrote:
@ronald-d-rogers <https://github.com/ronald-d-rogers> do you mean the
retrieval database?
—
Reply to this email directly, view it on GitHub
<#4 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA64EJ2ZHGVGWKELV4E6VW3U24UG5ANCNFSM5OBYI4NQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
from retro-pytorch.
lucidrains
commented on May 23, 2024
@ronald-d-rogers I don't know what their specific plans are. Just ran into someone working close to the eleuther founders who was also working on retro
from retro-pytorch.
enijkamp
commented on May 23, 2024
@lucidrains Yes, working on retro-fitting CodeGen, but may take a few more weeks:
https://mobile.twitter.com/arankomatsuzaki/status/1508246117351362560
from retro-pytorch.
ronald-d-rogers
commented on May 23, 2024
@lucidrains Yes, working on retro-fitting CodeGen, but may take a few more weeks: https://mobile.twitter.com/arankomatsuzaki/status/1508246117351362560
@enijkamp I think what you all are doing is great. A difference between this and other models though is that it's a two part system, one is the model and the other is the retrieval database. Have y'all thought about whether or not you'd open source the retrieval database as well? My understanding is that it would be quite large (~93TB for MassiveText which is 10.5TB on disk, so maybe ~8TB for The Pile?).
from retro-pytorch.
paraschopra
commented on May 23, 2024
@enijkamp great work with codegen. Looking forward to the open source version of RETRO.
from retro-pytorch.
Related Issues (20)
- Extra layer encoder_output_to_decoder_dim cause issue with distributed training HOT 2
- TrainingWrapper does not support line breaks HOT 8
- RuntimeError: Error in void faiss::gpu::GpuIndexIVFPQ::verifySettings_() HOT 3
- Double [CLS] token in the first doc chunk HOT 1
- Retro-fitting a pretrained model HOT 7
- Clarification on Architecture
- Scann vs faiss HOT 6
- 'NoneType' object is not callable HOT 1
- Is there any pre-trained RETRO model released yet? HOT 4
- Huggingface model
- I am revising the model to solve QA task.. HOT 1
- How to give Prompt to trained RETRO Model? HOT 6
- Why are there so many position embeddings? HOT 5
- Causal mask in Chunked Cross Attention
- Error # could not open .tmp/.index/knn.index for reading: No such file or directory
- Question-Answer Dataset Format ?
- AttributeError: module 'faiss' has no attribute 'GpuParameterSpace' HOT 2
- Question: residual connect after `ChunkedCrossAttention`? HOT 5
- Convert embedded tokens to English
- how to deal with the problem ,
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from retro-pytorch.