Tired of low-level languages? Ever wanted to infer a baby Llama 2 model in pure Julia? Great news – you can now do so at in under 300 lines of Julia.

This is a fork of Andrej's llama2.c which has been ported to (for now) a slightly hacky version of Julia. This README is heavily inspired by the Rust port llama.rs.

Don't want to read? Got ya back!

git clone https://github.com/juvi21/llama2.jl && cd llama2.jl && wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin && julia jl_helpers/install_pkg.jl && julia run.jl stories15M.bin tokenizer.bin

1. Grab Andrej's baby Llama2 (see the original instructions) pretrained on the TinyStories dataset:

   wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin
   wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories42M.bin
   wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.bin

2. Ensure you have the tokenizer binary - tokenizer.bin (if not, see tokenizer.py).

3. Run run.jl:

   Single-threaded:

   julia run.jl <model> <tokenizer> --temp [temperature]

   Multi-Threaded: In Progress
   CUDA: In Progress

On my current workstation, the performance is quite fast. However, I have been away visiting my parents for a few days, so I only had the opportunity to test it on one of my very first and less powerful station. More testing is coming soon! NOTE: I compiled llama2.c with the provided command in Andrej's README which is only the basic one to get started and not very optimized.

gcc -O3 -o run run.c -lm

Join the dark side and code in Julia. Contributions are highly encouraged!

Contribution Ideas:

• Make it faster.
• Add CUDA support.
• Introduce Multi-Threaded support.
• Cutom Prompt

@Midjourney