No output when using Baichuan2-7B-Chat with 2k input and int4 on XPU about bigdl HOT 8 CLOSED

We have tested with the above precisions and reproduced the issue not only for int4 but also fp32

We found this error might caused by the Baichuan's prompt format. The original format used by BigDL-LLM is "<human>{prompt} <bot>". After we changed the format to "<human>{prompt} <bot> 回答是:" to stress the output generation, the model could generate output as usual in all the above precisions

from bigdl.

@NovTi Let's find a better prompt template for Baichuan2.

from bigdl.

@NovTi Let's find a better prompt template for Baichuan2.

Found the official prompt template from link1 and link2, which is "<reserved_106> {prompt} <reserved_107>". Now the model can generate output at multiple precisions.

from bigdl.

When using the new prompt format, sometimes it will generate Chinese output for English prompt. For example, in the following two questions, first prompt is What is Deep Learning? but it answers in Chinese. The second example has no problem.

Prompt: What is Deep Learning?

Prompt: Tell me a story

from bigdl.

@plusbang Take a look. Seems the official template may not working well for Chinese input as well.
Can we still update this to the official template at this moment since no better templates could be found and we mainly focus on Chinese inputs for Baichuan?

from bigdl.

And also <reserved_106> may seem not that user friendly in the raw input string, but seems no better solutions so far?

from bigdl.

@plusbang Take a look. Seems the official template may not working well for Chinese input as well. Can we still update this to the official template at this moment since no better templates could be found and we mainly focus on Chinese inputs for Baichuan?

I feel we could use the official template, and add some comments to include the reference links and explanatory notes which demonstrates that other template may need for English prompt.

from bigdl.

Updated the prompt format: #10334
Closing this issue.

from bigdl.