This repo is my attempt at a rough implementation of nanoGPT. I train on a dataset of 30,000 unique Twitter usernames that mainly focus on tech and anime.
The primary motivation behind this project was to gain a deeper understanding of transformer models by implementing one from scratch. Training models on traditional datasets like Shakespeare's works sounded boring, so to make things interesting I decided to create my own dataset by scraping Twitter usernames, particularly those related to tech and anime.
Additionally, I needed a new Twitter username for myself anyway Lol
- PyTorch
- tinygrad
- JAX
Awesome Transformers Resources