This README describes steps for instruction fine-tuning of the Llama 13B model on the alpaca-gpt-4 dataset using HuggingFace Transformers and DeepSpeed. The script can be run on a single A100 GPU, for instance in Goggle Colab - just open the terminal and complete the steps listed below. Alternatively, you can run this on your choice of one or several GPUs.

• Python 3.6 or later.
• Basic familiarity with Python programming and virtual environments.

Creating a virtual environment is recommended to avoid conflicts with system-wide packages.

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

If cloning with HTTPS:

git clone https://github.com/agnedil/fine-tune-with-deepspeed.git

nvcc --version

If you need access to HuggingFace Hub, provide the access token after running this command:

huggingface-cli login

Install the required libraries using pip:

pip install accelerate peft bitsandbytes transformers trl datasets deepspeed

If you encounter a matrix multiplication error when running the script, you may want to downgrade the transformers package according to this issue:

!pip install git+https://github.com/huggingface/[email protected]

The fine-tuning script, script.py, is prepared for execution along with a config file ds_config.json. To run the script with DeepSpeed, execute the following command in your terminal in the same directory as the above two files (or adjust file paths respectively):

deepspeed --num_nodes 1 --num_gpus=1 script.py

Modify ds_config.json and replace m with a specific number of GPU nodes and n with a specific number of GPUs per node in the command below. For more details on running DeepSpeed, see the References section below.

deepspeed --num_nodes {m} --num_gpus={n} script.py

• DeepSpeed AI
• Integration of DeepSpeed with HuggingFace transformers
• Running DeepSpeed on a single GPU
• Miscrosoft DeepSpeed on GitHub