vihangd / alpaca-qlora Goto Github PK

I

Traceback (most recent call last):
File "/home/studio-lab-user/sagemaker-studiolab-notebooks/alpaca-qlora/finetune.py", line 419, in
fire.Fire(train)
File "/home/studio-lab-user/.conda/envs/studiolab/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/studio-lab-user/.conda/envs/studiolab/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/studio-lab-user/.conda/envs/studiolab/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/studio-lab-user/sagemaker-studiolab-notebooks/alpaca-qlora/finetune.py", line 347, in train
train_val = data["train"].train_test_split(
File "/home/studio-lab-user/.conda/envs/studiolab/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 545, in wrapper
out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
File "/home/studio-lab-user/.conda/envs/studiolab/lib/python3.9/site-packages/datasets/fingerprint.py", line 511, in wrapper
out = func(dataset, *args, **kwargs)
File "/home/studio-lab-user/.conda/envs/studiolab/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 4379, in train_test_split
raise ValueError(
ValueError: test_size=2000 should be either positive and smaller than the number of samples 2 or a float in the (0, 1) range

Maybe something to do with dataset .. using standard alpaca format
using >>
python finetune.py
--base_model 'openlm-research/open_llama_3b_600bt_preview'
--data_path '../datasets/dolly.json'
--num_epochs=3
--cutoff_len=512
--group_by_length
--output_dir='./dolly-lora-3b'
--lora_r=16
--lora_target_modules='[q_proj,v_proj]'

Where is the ../datasets/dolly.json file .. I would like to see this data how it look..
Any advise/ Thanks

qlory triaining using paged_adamw_32bit but you using paged_adamw_8bit here maybe a problem ?

Originally using these settings yields an error (because the dolly dataset is not in the default alpaca format)
python finetune.py --base_model 'openlm-research/open_llama_3b_v2' --data_path 'databricks/databricks-dolly-15k' --num_epochs=3 --cutoff_len=512 --group_by_length --output_dir='./dolly-lora-3b' --lora_r=16 --lora_target_modules='[q_proj,v_proj]'

I solve this by using the Dolly dataset in alpaca format, (published on the HF hub as 'c-s-ale/dolly-15k-instruction-alpaca-format'). This new training command seems to work now:
python finetune.py --base_model 'openlm-research/open_llama_3b_v2' --data_path 'c-s-ale/dolly-15k-instruction-alpaca-format' --num_epochs=3 --cutoff_len=512 --group_by_length --output_dir='./dolly-lora-3b' --lora_r=16 --lora_target_modules='[q_proj,v_proj]'

I notice that there are some differences compared to the artido/qlora repo. Why were the following code left out in this repo?

def find_all_linear_names(args, model):
    cls = bnb.nn.Linear4bit if args.bits == 4 else (bnb.nn.Linear8bitLt if args.bits == 8 else torch.nn.Linear)
    lora_module_names = set()
    for name, module in model.named_modules():
        if isinstance(module, cls):
            names = name.split('.')
            lora_module_names.add(names[0] if len(names) == 1 else names[-1])


    if 'lm_head' in lora_module_names: # needed for 16-bit
        lora_module_names.remove('lm_head')
    return list(lora_module_names)

https://github.com/artidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L221-L232

and

    for name, module in model.named_modules():
        if isinstance(module, LoraLayer):
            if args.bf16:
                module = module.to(torch.bfloat16)
        if 'norm' in name:
            module = module.to(torch.float32)
        if 'lm_head' in name or 'embed_tokens' in name:
            if hasattr(module, 'weight'):
                if args.bf16 and module.weight.dtype == torch.float32:
                    module = module.to(torch.bfloat16)
    return model

https://github.com/artidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L334-L344

Instead, you simply set the lora target modules to be ["q_proj", "v_proj"].

Thank you for any clarifications

Hello Vihang,

I am trying to find out if I can run efficiently inference of small number of parameters LLM on CPU hardware. Does your extension to 4 bit work also on CPU hardware? It seems to me that 4bit is only mentioned in when device is cuda.
Thanks in advance.

I have been trying to train a llama-7b as well as redpajama-3b using the official qlora repo.

The results however are not great. Most of the time I see a lot of repetitions or gibberish.

I have trained it on a small dataset of only 2000 rows but for 10 epochs.

Both train/eval loss were decreasing the whole time.

Only difference I see in their code and yours is that you don't attach lora to every linear layer as mentioned in the paper.

Hi, I noticed that you tried scaled rope. How could i use the scaled code for training a qlora with this?