This project focuses on the fine tuning of ChatGLM-6B-int4 model in different ways (freeze\embeding\PT\LoRA), and comparing the effect of different fine tuning methods on the large model, mainly for information extraction task, generation task, classification task, etc.
And if you fine tuning other version of ChatGLM-6B(like pf16), you need to upate the version corresponding to
configuration_chatglm.py
quantization.py
modeling_chatglm.py
tokenization_chatglm.py
test_modeling_chatglm.py
tokenization_chatglm.py
in https://huggingface.co/THUDM/chatglm-6b
The parameters of the original model are frozen. For example, only the layer behind the model can be trained.
The parameters of the final training are as follows：
trainable params: 81920 || all params: 3.356B || trainable%: 0.0024
be train layer: transformer.layers.23.input_layernorm.weight
be train layer: transformer.layers.23.input_layernorm.bias
be train layer: transformer.layers.23.post_attention_layernorm.weight
be train layer: transformer.layers.23.post_attention_layernorm.bias
be train layer: transformer.layers.24.input_layernorm.weight
be train layer: transformer.layers.24.input_layernorm.bias
be train layer: transformer.layers.24.post_attention_layernorm.weight
be train layer: transformer.layers.24.post_attention_layernorm.bias
be train layer: transformer.layers.25.input_layernorm.weight
be train layer: transformer.layers.25.input_layernorm.bias
be train layer: transformer.layers.25.post_attention_layernorm.weight
be train layer: transformer.layers.25.post_attention_layernorm.bias
be train layer: transformer.layers.26.input_layernorm.weight
be train layer: transformer.layers.26.input_layernorm.bias
be train layer: transformer.layers.26.post_attention_layernorm.weight
be train layer: transformer.layers.26.post_attention_layernorm.bias
be train layer: transformer.layers.27.input_layernorm.weight
be train layer: transformer.layers.27.input_layernorm.bias
be train layer: transformer.layers.27.post_attention_layernorm.weight
be train layer: transformer.layers.27.post_attention_layernorm.bias
Freeze the model entirely and train only the ebedding part of the model as one of the soft prompt ways.
The parameters of the final training are as follows：
trainable params: 0.53B || all params: 3.356B || trainable%: 15.9
be train layer: transformer.word_embeddings.weight
P Tuning P-tuning-V2 A soft prompt improvement,P-tuning-V2 is not only for the embedding layer, but continuous tokens are inserted into each layer, increasing the amount of change and interaction.
The parameters of the final training are as follows：
trainable params: 0.957B || all params: 4.312B || trainable%: 22.18
transformer.prefix_encoder.embedding.weight
transformer.prefix_encoder.trans.0.weight
transformer.prefix_encoder.trans.0.bias
transformer.prefix_encoder.trans.2.weight
transformer.prefix_encoder.trans.2.bias
LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during
adaptation instead, while keeping the pre-trained weights frozen.
\
Fine tuning the model in Google Colab pro with A100-40G,so you need to pip install somethings in Colab:
!pip install --upgrade tensorboard
!pip install --upgrade protobuf
!pip install transformers
!pip install sentencepiece
!pip install deepspeed
!pip install mpi4py
!pip install cpm_kernels
!pip install icetk
!pip install peft
!pip install tensorboard
!pip install tqdm
Freeze loss
embedding loss
PT loss
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent