Comments (4)
I can say that alpaca_lora_4bit is nice. It has no cyrillic issues (xturing has), and I had no problems to run it.
The hardware: RTX A4000
The dataset is same for both cases.
Differences between rulm (using rualpaca 7b) and alpaca_lora_4bit (using GPTQ v2 alpaca 7b):
- The speed of fine-tuning: 26.3 hours vs 7.1, 10 batches vs 50 batches (trying to use the nearly whole 16gb vram)
- Eval/loss: 1.214 vs 1.064
- Inference: 15.2gb vs 5.7gb, the speed is 3-5 tokens vs 13 tokens
- Quality of speech: rualpaca is definitely better for russian speech. alpaca_lora_4bit often tries to use other languages or generates a code:
For now, quantized model is more prefferable for me because I can get results faster - both inference and fine-tuning. Probably full training without lora will be also affordable on such small model.
It would be great if rulm will start to support alpaca_lora_4bit - 7b inference and fine_tuning can be fitted in 8gb vram, it's awesome result.
from rulm.
Hi! I'm not sure how you are supposed to train models in 4-bits. AFAIK only 8-bit training is commonly supported by now, although I haven't seen many full 8-bit fine tunes.
Usually, the easiest option is to train adapters, for instance, LoRA. In that case, you can quantize a model into 8 bits with mixed RTN quantization. It is natively supported in the transformers package. After that, you freeze the original weights and train only new weights.
from rulm.
Hi! This project does support 4bit training - https://github.com/johnsmith0031/alpaca_lora_4bit
For now (started yesterday) I'm trying to combine rulm and alpaca_lora_4bit and check the outcome. As far as I can see, the fine-tune process will last less time. Unfortunately, alpaca_lora_4bit has no wandb integration. I'll write updates in this issue.
Also bitsandbytes announced 4bit support - https://www.reddit.com/r/LocalLLaMA/comments/13ahz60/bitsandbytes_4bit_finetuning_30b65b_llama_models/
Probably it's the easier way to get quantized models fine-tuning.
from rulm.
Thank you for sharing your results!
I'll definitely try to use code from alpaca_lora_4bit, though it seems to be highly experimental.
from rulm.
Related Issues (20)
- Error while trying to train llama_13b following the guide HOT 1
- Is there any way to increase speed? HOT 2
- I can't start, what is the error, please tell me HOT 2
- Is it possible to run under windows+ python + CUDA? HOT 3
- Не удалось получить ожидаемые результаты при обучении HOT 8
- Неправильно форматирование prompt'а? HOT 2
- Воспроизведение результатов для Saiga2 HOT 2
- Why results are much worse on V100? HOT 2
- Модели путают склонения, падежи и т.д. HOT 5
- Проблема запуска ggml версии HOT 2
- Проблема с режимами fine tuning
- Пример Collab файнтюн. Ошибка на этапе скачивания базовой модели. HOT 3
- Mistral 7B - лучше в русском чем saiga2_7b HOT 1
- LLaVA 7B/13B - Будущая русская GPT4V?
- Двойной EOS-токен в скрипте генерации датасета
- convert_to_native.py 70b support
- Qwen models
- Posting issues rulm
- Fine tuning with adapters HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rulm.