尝试在12G卡上训练 python qlora.py --model_name="chinese_alpaca" --model_name_or_path="./model_hub/chinese-alpaca-7b" --trust_remote_code=False --dataset="msra" --source_max_len=128 --target_max_len=64 --do_train --save_total_limit=1 --padding_side="right" --per_device_train_batch_size=8 --do_eval --bits=4 --save_steps=10 --gradient_accumulation_steps=1 --learning_rate=1e-5 --output_dir="./output/alpaca/" --lora_r=8 --lora_alpha=32
出错：
File "/mnt/data1ts/llm/training/qlora-chinese-LLM/qlora.py", line 1012, in
train()
File "/mnt/data1ts/llm/training/qlora-chinese-LLM/qlora.py", line 973, in train
train_result = trainer.train(resume_from_checkpoint=checkpoint_dir)

result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1536x4096 and 1x8388608)
可能是什么原因导致? 谢谢。

麻烦问下，如何把chatgml和lora合并到一个模型

ValueError: You can't train a model that has been loaded in 8-bit precision on a different device than the one you're training on.

或者报错ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices.

(gh_qlora-chinese-LLM) ub2004@ub2004-B85M-A0:~/llm_dev/qlora-chinese-LLM$ python3 qlora.py --model_name="chatglm" --model_name_or_path="/data-ssd-1t/hf_model/chatglm-6b" --trust_remote_code=True --dataset="msra" --source_max_len=128 --target_max_len=64 --do_train --save_total_limit=1 --padding_side="left" --per_device_train_batch_size=8 --do_eval --bits=4 --save_steps=10 --gradient_accumulation_steps=1 --learning_rate=1e-5 --output_dir="./output/chatglm-6b/" --lora_r=8 --lora_alpha=32

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /home/ub2004/anaconda3/envs/gh_qlora-chinese-LLM/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/ub2004/anaconda3/envs/gh_qlora-chinese-LLM/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/ub2004/anaconda3/envs/gh_qlora-chinese-LLM/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
CUDA SETUP: Loading binary /home/ub2004/anaconda3/envs/gh_qlora-chinese-LLM/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
loading base model /data-ssd-1t/hf_model/chatglm-6b...
Traceback (most recent call last):
File "/home/ub2004/llm_dev/qlora-chinese-LLM/qlora.py", line 1011, in
train()
File "/home/ub2004/llm_dev/qlora-chinese-LLM/qlora.py", line 836, in train
model = get_accelerate_model(args, checkpoint_dir)
File "/home/ub2004/llm_dev/qlora-chinese-LLM/qlora.py", line 375, in get_accelerate_model
model = model_class[args.model_name].from_pretrained(
File "/home/ub2004/llm_dev/qlora-chinese-LLM/transformers/src/transformers/models/auto/auto_factory.py", line 479, in from_pretrained
return model_class.from_pretrained(
File "/home/ub2004/llm_dev/qlora-chinese-LLM/transformers/src/transformers/modeling_utils.py", line 2819, in from_pretrained
raise ValueError(
ValueError:
Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom
device_map to from_pretrained. Check
https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
for more details.

(gh_qlora-chinese-LLM) ub2004@ub2004-B85M-A0:~/llm_dev/qlora-chinese-LLM$

同样的peft版本，但是会报错。您这边的环境能跑通么？
另外，删去这个参数就可以跑通了

不知道大佬有没有遇到ValueError: paged_adamw_32bit is not a valid OptimizerNames这个错误

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

请问我遇到了这个问题，如何解决呢？

我训练的时候占比高达0.86，target_modules为'dense', 'dense_4h_to_h', 'dense_h_to_4h', 'query_key_value'