randaller / llama-cpu Goto Github PK

Using BFLOAT16 and 7B model. Fast to read and low mem but error before answer.
No solved replacing previous files (generation y model) on "llama" folder.

prompt> the meaning of the life is
0%| | 0/256 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/user/IA/ll-cpu3/llama-cpu/example-chat.py", line 109, in
fire.Fire(main)
...
...
File "/home/user/IA/ll-cpu3/llama-cpu/llama/model.py", line 109, in forward
xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: expected scalar type Float but found BFloat16
0%| | 0/256 [00:02<?, ?it/s]

I have totally 56 CPUs, but the max cpu usage is below 50% during inference. How to increase the usage of CPUs?

When I run python3 merge-weights.py --input_dir /home//llama-cpu/ --model_size 7B I receive the following error:

Traceback (most recent call last):
  File "merge-weights.py", line 168, in <module>
    main()
  File "merge-weights.py", line 161, in main
    write_model(
  File "merge-weights.py", line 63, in write_model
    state_dict |= {
TypeError: unsupported operand type(s) for |=: 'dict' and 'dict'

In the README I understood that it is necessary to run example-chat-bfloat16.py

any idea why?