This is an experimental repository for me to learn more about Transformers, GPT, and RLHF. There is no RLHF in this repository yet.
poetry install
poetry run pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
- use dataset in https://github.com/tysam-code/hlb-gpt
- refactor
train_sort
- support multi-GPUs