Fine tuning HuggingFace CausalLMs to generate text resembling J. R. R. Tolkien's writing.
This repo contains the code to fine tune a causalLM of your choice as well as easily generating your own text.
Personally, I tested this using the the models EleutherAI/gpt-neo-125M
and facebook/opt-350M
. In theory, any other hugging face CausalLM can be used, but given my memeory constrains (32GB RAM w/ GTX 3090) I was limited to relatively smaller models. For the most part, larger and more recently released models should produce better results.
While I only fine-tuned on Tolkien's writings, any text can be used. For the best results, I suggest splitting text into roughly paragraph length.
To fine-tune a model:
bin/train.sh --model_name model_name
model_name defaults to facebook/opt-350M
. Run bin/train.sh -h
for a list of other parameters.
To generate text:
bin/gen.sh --model_name model_name --model_dir model_dir
Where model_name
is the hugging face pre trained model, and model_dir
is the fine tuned model.
model_name
defaults to facebook/opt-350M