loading ggmlv3.q4 and q4_K_M about llama2-webui HOT 6 CLOSED

Hi, can you show your .env file. CUDA not found. means you are trying to load Llama2, gptq models.
If you are loading ggml models, it should print out like this .
Make sure you have set LLAMA_CPP = True in .env file.

from llama2-webui.

hey, thanks for the quick reply. I used the example for llama-2-7b-chat.ggmlv3.q4_0.bin
This is my .env file:

MODEL_PATH = "/home/user/project/models/llama-2-7b-chat.ggmlv3.q4_0.bin"
LOAD_IN_8BIT = False
LOAD_IN_4BIT = True
LLAMA_CPP = True

MAX_MAX_NEW_TOKENS = 2048
DEFAULT_MAX_NEW_TOKENS = 1024
MAX_INPUT_TOKEN_LENGTH = 4000

DEFAULT_SYSTEM_PROMPT = "\
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\
"

from llama2-webui.

mmh, I didn't really change anything, mostly reloaded my venv (but which I also did before) and now it says it is running on CPU with llama.cpp. Though now it also does not find the model and seemingly doesn't pick up the path properly from .env.

from llama2-webui.

Now it seems to work. Apparently for a venv, it only picks up changes to the .env after exiting the venv/environment and reloading it?

from llama2-webui.

It is wired. The '.env' should be loaded every time you run app.py.

from llama2-webui.

@step21 welcome contributing your benchmark performance here.

from llama2-webui.