Comments (3)
@JoshuaeKaiser Hi, llama sampling is not idempotent if you setup topk topp and nonzero temperature.ie. your output will never be same for same parameters(except that you set temperature to 0 but not sure if the backend has implemeted for this). could you provide an example for both with temp=0.1 topk=1 topp=0.1? which may generate more stable output.
additionally for vicuna please upgrade to v1.1 and use the new instruct template as described here. this allow you to remove stopSequence
from llama-node.
@hlhr202 Thank you for your detailed reply. I have managed to track down the issue, I was using the master prompt template the original docks specified instead of what your new example states shown bellow.
const prompt = A chat between a user and an assistant. USER: ${template} ASSISTANT:
;
One observation, the new example does not specify the "stopSequence" property, which I have come to find is crucial when specifiying that the model stop, so "stopSequence": "USER:" should probably be added to that example, otherwise the model continues forever until it hits maxium output.
Couple of questions if you will indulge me. What do these properties denote when passed as config. I have added coments regarding my assumptions on some of them.
nCtx: 1024, //Maximum input context size of the prompt
nParts: -1,
seed: 0,
f16Kv: false,
logitsAll: false,
vocabOnly: false,
useMlock: false,
embedding: true, //Enables embedding mode so that you can generate embeddings using the specified methods
useMmap: true,
nThreads: 4,
nTokPredict: 2048, //Maximum number of tokens the model will output
topK: 40,
topP: 0.1,
One further question. For running multiple instances of the model for example sticking the model behind a websocket api, do you just recomend calling the LLAMA constructor on every request and loading the model each time like below? I ask because this obviously adds time to the inference output. Wondering what the best approach is here to increase the speed at which inference starts
const llama = new LLama(LLamaCpp);
llama.load({path: model, ...config})
from llama-node.
@hlhr202 Thank you for your detailed reply. I have managed to track down the issue, I was using the master prompt template the original docks specified instead of what your new example states shown bellow.
const prompt =
A chat between a user and an assistant. USER: ${template} ASSISTANT:
;One observation, the new example does not specify the "stopSequence" property, which I have come to find is crucial when specifiying that the model stop, so "stopSequence": "USER:" should probably be added to that example, otherwise the model continues forever until it hits maxium output.
Couple of questions if you will indulge me. What do these properties denote when passed as config. I have added coments regarding my assumptions on some of them.
nCtx: 1024, //Maximum input context size of the prompt nParts: -1, seed: 0, f16Kv: false, logitsAll: false, vocabOnly: false, useMlock: false, embedding: true, //Enables embedding mode so that you can generate embeddings using the specified methods useMmap: true, nThreads: 4, nTokPredict: 2048, //Maximum number of tokens the model will output topK: 40, topP: 0.1,
One further question. For running multiple instances of the model for example sticking the model behind a websocket api, do you just recomend calling the LLAMA constructor on every request and loading the model each time like below? I ask because this obviously adds time to the inference output. Wondering what the best approach is here to increase the speed at which inference starts
const llama = new LLama(LLamaCpp);
llama.load({path: model, ...config})
@JoshuaeKaiser
Since I havent implemented parallel inference yet, if you want to serve llama-node for multi-inference concurrency purpose, you should create multiple llama instances(which consumes n x memory). but if you are just serve for one inference session, you can reuse the loaded model. which means you can just load model once, and do multiple inferences later.
from llama-node.
Related Issues (20)
- Unsupported file version 101 when loading rwkv
- Should bring details in error message
- δΈζδΉ±η
- Use custom tokenizer
- Discord Access HOT 1
- Llama2 quantized q5_1 HOT 1
- fatal runtime error: Rust cannot catch foreign exceptions HOT 5
- [Error: Failed to get embeddings] { code: 'GenericFailure' }
- Segmentation fault HOT 1
- GPU version build not using GPU HOT 11
- Support for passing grammars
- Can't run example on llama-2-13b-chat q4_0 HOT 2
- The requested module 'llama-node/dist/llm/llama-cpp.js' does not provide an export named 'LoadConfig'
- napi: not found on debian bullseye HOT 5
- Time printings are gone
- Instructions for use with Electron
- cannot use RWKV models HOT 16
- GGUF support?
- Alternative? HOT 1
- What files are compatible?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llama-node.