Comments (8)
We have tested with the above precisions and reproduced the issue not only for int4 but also fp32
We found this error might caused by the Baichuan's prompt format. The original format used by BigDL-LLM is "<human>{prompt} <bot>"
. After we changed the format to "<human>{prompt} <bot> 回答是:"
to stress the output generation, the model could generate output as usual in all the above precisions
from bigdl.
@NovTi Let's find a better prompt template for Baichuan2.
from bigdl.
@NovTi Let's find a better prompt template for Baichuan2.
Found the official prompt template from link1 and link2, which is "<reserved_106> {prompt} <reserved_107>"
. Now the model can generate output at multiple precisions.
from bigdl.
When using the new prompt format, sometimes it will generate Chinese output for English prompt. For example, in the following two questions, first prompt is What is Deep Learning?
but it answers in Chinese. The second example has no problem.
Prompt: What is Deep Learning?
from bigdl.
@plusbang Take a look. Seems the official template may not working well for Chinese input as well.
Can we still update this to the official template at this moment since no better templates could be found and we mainly focus on Chinese inputs for Baichuan?
from bigdl.
And also <reserved_106> may seem not that user friendly in the raw input string, but seems no better solutions so far?
from bigdl.
@plusbang Take a look. Seems the official template may not working well for Chinese input as well. Can we still update this to the official template at this moment since no better templates could be found and we mainly focus on Chinese inputs for Baichuan?
I feel we could use the official template, and add some comments to include the reference links and explanatory notes which demonstrates that other template may need for English prompt.
from bigdl.
Updated the prompt format: #10334
Closing this issue.
from bigdl.
Related Issues (20)
- Facing issue when python3.9 -m pip install bigdl-chronos[pytorch]==2.4.0 HOT 4
- Can ipex-llm-0.43.1 run on Centos7.9? HOT 7
- I have Ultra 9 185h, which (IPEX or OpenVINO) should I choose for LLM and Stable Diffusion?
- Native API returns: -2 (PI_ERROR_DEVICE_NOT_AVAILABLE) HOT 25
- Looking for a workaround to install IPEX-LLM on Windows with an Intel GPU but running with a CPU and not running with a GPU HOT 4
- microsoft/Florence-2-large Unable to perform Object Detection on Intel Arc GPU 770 HOT 4
- running bigdl inference while click the mouse will cause GPU hang issue HOT 3
- Modify the benchmark output file name with hours, minutes information HOT 2
- Add multi GPU support in the AutoModelForCausalLM.load_low_bit API. HOT 1
- MiniCPM-V-2 run error on Windows MTL iGPU HOT 1
- ImportError: undefined symbol: iJIT_NotifyEvent on 2-ARC GPU HOT 1
- Qwen1.5-GPTQ-Int4 inference error HOT 4
- inference error: mistral and codellama have issue 'object has no attribute '_has_non_default_generation_parameters' HOT 1
- Incompatible with Streamlit HOT 3
- run "python setup.py bdist_wheel" error HOT 3
- vLLM freezes with gpu-memory-utilization > 0.55 HOT 3
- ollama MTL iGPU issue HOT 7
- Qwen1.5-A2.7B cannot be offload on iGPU HOT 3
- CodeQwen1.5-7B-Chat performance gap between pytorch and gguf
- Segmentation fault when running Deepspeed-AutoTP HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigdl.