Version 0.0.34 gives drastic different ouput when paired with vicuna. about llama-node HOT 3 OPEN

@JoshuaeKaiser Hi, llama sampling is not idempotent if you setup topk topp and nonzero temperature.ie. your output will never be same for same parameters(except that you set temperature to 0 but not sure if the backend has implemeted for this). could you provide an example for both with temp=0.1 topk=1 topp=0.1? which may generate more stable output.
additionally for vicuna please upgrade to v1.1 and use the new instruct template as described here. this allow you to remove stopSequence

from llama-node.

@hlhr202 Thank you for your detailed reply. I have managed to track down the issue, I was using the master prompt template the original docks specified instead of what your new example states shown bellow.

const prompt = A chat between a user and an assistant. USER: ${template} ASSISTANT:;

One observation, the new example does not specify the "stopSequence" property, which I have come to find is crucial when specifiying that the model stop, so "stopSequence": "USER:" should probably be added to that example, otherwise the model continues forever until it hits maxium output.

Couple of questions if you will indulge me. What do these properties denote when passed as config. I have added coments regarding my assumptions on some of them.

nCtx: 1024, //Maximum input context size of the prompt
nParts: -1,
seed: 0,
f16Kv: false,
logitsAll: false,
vocabOnly: false,
useMlock: false,
embedding: true, //Enables embedding mode so that you can generate embeddings using the specified methods
useMmap: true,

nThreads: 4,
nTokPredict: 2048, //Maximum number of tokens the model will output
topK: 40,
topP: 0.1,

One further question. For running multiple instances of the model for example sticking the model behind a websocket api, do you just recomend calling the LLAMA constructor on every request and loading the model each time like below? I ask because this obviously adds time to the inference output. Wondering what the best approach is here to increase the speed at which inference starts

const llama = new LLama(LLamaCpp);

llama.load({path: model, ...config})

from llama-node.

@hlhr202 Thank you for your detailed reply. I have managed to track down the issue, I was using the master prompt template the original docks specified instead of what your new example states shown bellow.

const prompt = A chat between a user and an assistant. USER: ${template} ASSISTANT:;

One observation, the new example does not specify the "stopSequence" property, which I have come to find is crucial when specifiying that the model stop, so "stopSequence": "USER:" should probably be added to that example, otherwise the model continues forever until it hits maxium output.

Couple of questions if you will indulge me. What do these properties denote when passed as config. I have added coments regarding my assumptions on some of them.

nCtx: 1024, //Maximum input context size of the prompt
nParts: -1,
seed: 0,
f16Kv: false,
logitsAll: false,
vocabOnly: false,
useMlock: false,
embedding: true, //Enables embedding mode so that you can generate embeddings using the specified methods
useMmap: true,

nThreads: 4,
nTokPredict: 2048, //Maximum number of tokens the model will output
topK: 40,
topP: 0.1,

One further question. For running multiple instances of the model for example sticking the model behind a websocket api, do you just recomend calling the LLAMA constructor on every request and loading the model each time like below? I ask because this obviously adds time to the inference output. Wondering what the best approach is here to increase the speed at which inference starts

const llama = new LLama(LLamaCpp);

llama.load({path: model, ...config})

@JoshuaeKaiser
Since I havent implemented parallel inference yet, if you want to serve llama-node for multi-inference concurrency purpose, you should create multiple llama instances(which consumes n x memory). but if you are just serve for one inference session, you can reuse the loaded model. which means you can just load model once, and do multiple inferences later.

from llama-node.