lightonai / akronomicon Goto Github PK

It now redirects to https://muse.lighton.ai/. The archive shows the last time it was working correctly was on March 4.

Image GPT-L compute is 248 PF-days, same as Image GPT-XL.

I find it implausible that the amount of compute is the same as the larger model.

Here is an alternate (not 100% reliable) source which estimates 6.5e21 FLOPS total
https://www.lesswrong.com/posts/wfpdejMWog4vEDLDg/ai-and-compute-trend-isn-t-predictive-of-what-is-happening

Training compute estimates for models in this database are not always stated in the publications for the models. For example, the AI21 Jurassic whitepaper does not report how much compute was used to train the models, but the database says 3708 petaflop-days (by the way, the units also are not clear just from the JSON files - but that's another issue). I'm not sure what the source or calculation for this number is.

Given that training compute estimates involve some assumptions and uncertainty, it would be good to include some explanation of how the compute was calculated. I realise it might be too cumbersome to put this in the database files themselves, but it could be e.g. in a README in each folder of the database, and/or in the commit messages when a specific database file is updated.

Would there be interest in an option to filter the leaderboard to open source models only? I feel like models like T5, GPT-J, etc which have been released deserve being applauded for that fact. And as a practical matter, for researchers who want to apply these models in their work it takes some searching to find the largest open source models of each type.

GPT-NeoX 20B is a new language model by EleutherAI trained on the Pile. It is a decoder model that is competent at both English and generating code.

The training code can be found here and the model weights will be released next week. It was trained using PyTorch and DeepSpeed on 96 A100 @ CoreWeave. You can find the compute states here... sorry, you'll have to do the math for the total compute yo