How to use exllama v2 with local GPTQ file ? about h2ogpt HOT 3 CLOSED

With latest builds of llama.cpp, there's no need for exllama v2 anymore in terms of speed or stability.

Some have asked for exllama v2 support, and it might be interesting last month, but after new llama.cpp I no longer will do this as llama.cpp is as fast.

exllama 0.0.18 used in h2oGPT is pretty old, and if it's broken I'll just remove it instead of trying to support exllama v2, it requires too many changes. They didn't make it easy to use.

from h2ogpt.

Yes, for path you just do --base_model=llama --model_path_llama=<path> .

https://github.com/h2oai/h2ogpt/blob/main/docs/FAQ.md#gguf--ggml

We check for the actual path (e.g. local or absolute) or if it is inside --llamacpp_path which defaults to ./llamacpp_path folder where we download stuff too.

from h2ogpt.

@pseudotensor with that run command python generate.py --base_model=TheBloke/Mistral-7B-Instruct-v0.2-GGUF --prompt_type=mistral --max_seq_len=4096

Can I specify the path for the model file ?
Thanks

from h2ogpt.