This repo is my attempt at a rough implementation of nanoGPT. I train on a dataset of 30,000 unique Twitter usernames that mainly focus on tech and anime.

The primary motivation behind this project was to gain a deeper understanding of transformer models by implementing one from scratch. Training models on traditional datasets like Shakespeare's works sounded boring, so to make things interesting I decided to create my own dataset by scraping Twitter usernames, particularly those related to tech and anime.

Additionally, I needed a new Twitter username for myself anyway Lol

• PyTorch
• tinygrad
• JAX

Awesome Transformers Resources

• Formal Algorithms for Transformers
• The Illustrated Transformer
• Master Positional Encoding
• Let's build the GPT Tokenizer
• Let's build GPT: from scratch, in code, spelled out.