This README describes steps for instruction fine-tuning of the Llama 13B model
on the alpaca-gpt-4
dataset using HuggingFace Transformers and DeepSpeed. The script can be run on a single A100 GPU, for instance in Goggle Colab - just open the terminal and complete the steps listed below. Alternatively, you can run this on your choice of one or several GPUs.
- Python 3.6 or later.
- Basic familiarity with Python programming and virtual environments.
Creating a virtual environment is recommended to avoid conflicts with system-wide packages.
python -m venv venv
source venv/bin/activate # On Windows, use `venv\Scripts\activate`
If cloning with HTTPS:
git clone https://github.com/agnedil/fine-tune-with-deepspeed.git
nvcc --version
If you need access to HuggingFace Hub, provide the access token after running this command:
huggingface-cli login
Install the required libraries using pip:
pip install accelerate peft bitsandbytes transformers trl datasets deepspeed
If you encounter a matrix multiplication error when running the script, you may want to downgrade the transformers package according to this issue:
!pip install git+https://github.com/huggingface/[email protected]
The fine-tuning script, script.py, is prepared for execution along with a config file ds_config.json. To run the script with DeepSpeed, execute the following command in your terminal in the same directory as the above two files (or adjust file paths respectively):
deepspeed --num_nodes 1 --num_gpus=1 script.py
Modify ds_config.json and replace m with a specific number of GPU nodes and n with a specific number of GPUs per node in the command below. For more details on running DeepSpeed, see the References section below.
deepspeed --num_nodes {m} --num_gpus={n} script.py