Baichuan2-7B takes more memory than chatglm3-6B on MTL 16GB device, need to optimize VRAM of Baichuan2-7B about bigdl HOT 4 CLOSED

You can use fp16 model to save more memory, convert model to fp16 before to xpu, like:

model = model.half().to('xpu')

This can save 700MB for 1024 tokens, 1400MB for 2048 tokens.

from bigdl.

If you can change the modeling file, you can empty xpu cache after https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat/blob/ea66ced17780ca3db39bc9f8aa601d8463db3da5/modeling_baichuan.py#L698
Like:

if hidden_states.device.type == "xpu" and hidden_states.size(1) != 1:
    torch.xpu.empty_cache()

#10317 is working on the empty, but it's still testing. This can decrease the peak memory about 600MB.

Another unused memory is the sin_cached and cos_cached in https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat/blob/main/modeling_baichuan.py#L109, about 153MB cached on cpu. You can remove them.

from bigdl.

It work, thanks!

from bigdl.

Thanks Qiu,Xin!

from bigdl.