aesthisia / llminator Goto Github PK

Gradio based tool to run opensource LLM models directly from Huggingface

License: MIT License

Python 82.72% CSS 0.64% CMake 4.66% Makefile 4.83% HTML 4.82% Dockerfile 2.33%

chatbot cpu cuda gguf gradio huggingface langchain llama-cpp-python llamacpp llm llm-inference ollama openai python safetensors model-conversion opensource streaming websockets-chat

llminator's Introduction

LLMinator: Run & Test LLMs directly from HuggingFace

Gradio based tool with integrated chatbot to locally run & test LLMs directly from HuggingFace.

An easy-to-use tool made with Gradio, LangChain, and Torch.

⚡ Features

Context-aware Streaming Chatbot.
Inbuilt code syntax highlighting.
Load any LLM repo directly from HuggingFace.
Supports both CPU & CUDA modes.
Enable LLM inference with llama.cpp using llama-cpp-python
Convert models(Safetensors, pt to gguf etc)
Customize LLM inference parameters(n_gpu_layers, temperature, max_tokens etc)
Real-time text generation via websockets, enabling seamless integration with different frontend frameworks.

🚀 Installation

To use LLMinator, follow these simple steps:

Clone the LLMinator repository from GitHub & install requirements

```
git clone https://github.com/Aesthisia/LLMinator.git
cd LLMinator
pip install -r requirements.txt
```

Build LLMinator with llama.cpp:

Using make:
- On Linux or MacOS:
```
make
```
- On Windows:
  1. Download the latest fortran version of w64devkit.
  2. Extract w64devkit on your pc.
  3. Run w64devkit.exe.
  4. Use the cd command to reach the LLMinator folder.
  5. From here you can run:
```
make
```
Using CMake:
```
mkdir build
cd build
cmake ..
```

Launch LLMinator on browser

Run the LLMinator tool using the command python webui.py.
Access the web interface by opening the http://127.0.0.1:7860 in your browser.
Start interacting with the chatbot and experimenting with LLMs!

Checkout this youtube video to follow installation steps

Command line arguments

Argument Command	Default	Description
--host	127.0.0.1	Host or IP address on which the server will listen for incoming connections
--port	7860	Launch gradio with given server port
--share	False	This generates a public shareable link that you can send to anybody

Connect to WebSocket for generation

Connect to ws://localhost:7861/ for real-time text generation. Submit prompts and receive responses through the websocket connection.

Integration with Frontends:

The provided example/index.html demonstrates basic usage of text generation through websocket connection. You can integrate it with any frontend framework like React.js

Installation and Development Tips

Python Version

Compatible Versions: This project is compatible with Python versions 3.8+ to 3.11. Ensure you have one of these versions installed on your system. You can check your Python version by running python --version or python3 --version in your terminal.

Cmake and C Compiler

Cmake Dependency: If you plan to build the project using Cmake, make sure you have Cmake installed.
C Compiler: Additionally, you'll need a C compiler such as GCC. These are typically included with most Linux distributions. You can check this by running gcc --version in your terminal. Installation instructions for your specific operating system can be found online.

Visual Studio Code

Visual Studio Installer: If you're using Visual Studio Code for development, you'll need the C++ development workload installed. You can achieve this through the Visual Studio Installer

GPU Acceleration (CUDA):

CUDA Installation: To leverage GPU acceleration, you'll need CUDA installed on your system. Download instructions are available on the NVIDIA website.
Torch Compatibility: After installing CUDA, confirm CUDA availability with torch.cuda.is_available(). When using a GPU, ensure you follow the project's specific llama-cpp-python installation configuration for CUDA support.

Reporting Issues:

If you encounter any errors or issues, feel free to file a detailed report in the project's repository. We're always happy to help! When reporting an issue, please provide as much information as possible, including the error message, logs, the steps you took, and your system configuration. This makes it easier for us to diagnose and fix the problem quickly.

🤝 Contributions

We welcome contributions from the community to enhance LLMinator further. If you'd like to contribute, please follow these guidelines:

Fork the LLMinator repository on GitHub.
Create a new branch for your feature or bug fix.
Test your changes thoroughly.
Submit a pull request, providing a clear description of the changes you've made.

Reach out to us: [email protected]

llminator's People

Contributors

Stargazers

Watchers

Forkers

subhanshu0027 cygwynd techthiyanes orefaleoluwayinka zeroxclem hercules261188 yacineali74 mdwoicke sontoriyama supersonic13 nymbo xjohnxjohn gopalgtm

llminator's Issues

ImportError: Could not import llama-cpp-python library

Running webui.py after fixing #42 yields:

This is also on a fresh system, after installing requirements and make-ing without errors. Also tried in a fresh conda env, and on the base python default in the ubuntu machine.

pip install llama-cpp-python says it's already installed. Any thoughts? Thanks

no errors, but not running

Hi there,

following the readme instructions I cloned and make the repo. Below are the logs and information, I hope useful.

The problem: The server is not running. It exits in the Python prompt

Expected: to have it running on localhost 7860

Any help will be appreciated.

Thank you

Node: I run this in vevn

Versions

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

uname -r
6.5.0-35-generic

python3 --version
Python 3.10.12
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
pip --version
pip 22.0.2

Make

mkdir -p src/llama_cpp
git clone https://github.com/ggerganov/llama.cpp src/llama_cpp
Cloning into 'src/llama_cpp'...
remote: Enumerating objects: 25832, done.
remote: Counting objects: 100% (25831/25831), done.
remote: Compressing objects: 100% (7528/7528), done.
remote: Total 25832 (delta 18441), reused 25221 (delta 18077), pack-reused 1
Receiving objects: 100% (25832/25832), 46.28 MiB | 14.76 MiB/s, done.
Resolving deltas: 100% (18441/18441), done.
cd src/llama_cpp && \
	/home/ivo/Projects/Python/LLMinator/bin/pip3 install -r requirements.txt
Collecting numpy~=1.24.4
  Downloading numpy-1.24.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 40.7 MB/s eta 0:00:00
Collecting sentencepiece~=0.2.0
  Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 39.1 MB/s eta 0:00:00
Collecting transformers<5.0.0,>=4.40.1
  Downloading transformers-4.41.2-py3-none-any.whl (9.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.1/9.1 MB 49.8 MB/s eta 0:00:00
Collecting gguf>=0.1.0
  Downloading gguf-0.6.0-py3-none-any.whl (23 kB)
Collecting protobuf<5.0.0,>=4.21.0
  Downloading protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.6/294.6 KB 26.5 MB/s eta 0:00:00
Requirement already satisfied: torch~=2.1.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from -r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.1.2)
Collecting huggingface-hub<1.0,>=0.23.0
  Downloading huggingface_hub-0.23.2-py3-none-any.whl (401 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.7/401.7 KB 30.0 MB/s eta 0:00:00
Requirement already satisfied: tqdm>=4.27 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (4.66.4)
Collecting tokenizers<0.20,>=0.19
  Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 45.2 MB/s eta 0:00:00
Requirement already satisfied: requests in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (2.32.3)
Collecting safetensors>=0.4.1
  Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 40.9 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (23.2)
Requirement already satisfied: pyyaml>=5.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (6.0.1)
Requirement already satisfied: filelock in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (3.14.0)
Collecting regex!=2019.12.17
  Downloading regex-2024.5.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (775 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 775.1/775.1 KB 29.5 MB/s eta 0:00:00
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: fsspec in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2024.5.0)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (8.9.2.26)
Requirement already satisfied: typing-extensions in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (4.12.0)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.0.106)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: networkx in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (3.3)
Requirement already satisfied: jinja2 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (3.1.4)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.3.1)
Requirement already satisfied: triton==2.1.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.1.0)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.18.1)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (11.4.5.107)
Requirement already satisfied: sympy in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (1.12.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (10.3.2.106)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.1.105)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (12.5.40)
Requirement already satisfied: MarkupSafe>=2.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from jinja2->torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (2.1.5)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (2024.2.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from requests->transformers<5.0.0,>=4.40.1->-r ./requirements/requirements-convert-legacy-llama.txt (line 3)) (3.7)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages (from sympy->torch~=2.1.1->-r ./requirements/requirements-convert-hf-to-gguf.txt (line 2)) (1.3.0)
Installing collected packages: sentencepiece, safetensors, regex, protobuf, numpy, huggingface-hub, gguf, tokenizers, transformers
  Attempting uninstall: numpy
    Found existing installation: numpy 1.26.4
    Uninstalling numpy-1.26.4:
      Successfully uninstalled numpy-1.26.4
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.21.1
    Uninstalling huggingface-hub-0.21.1:
      Successfully uninstalled huggingface-hub-0.21.1
Successfully installed gguf-0.6.0 huggingface-hub-0.23.2 numpy-1.24.4 protobuf-4.25.3 regex-2024.5.15 safetensors-0.4.3 sentencepiece-0.2.0 tokenizers-0.19.1 transformers-4.41.2
mkdir -p src/quantized_model
echo "py_cmd = /home/ivo/Projects/Python/LLMinator/bin/python" >> configs/config.ini

webui

python3 webui.py 
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/huggingface_hub/file_download.py:1194: UserWarning: `local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`.
For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.
  warnings.warn(
README.md: 100%|███████████████████████████████████████████████████████████████████| 6.72k/6.72k [00:00<00:00, 15.6MB/s]
generation_config.json: 100%|███████████████████████████████████████████████████████████| 132/132 [00:00<00:00, 713kB/s]
LICENSE: 100%|█████████████████████████████████████████████████████████████████████| 7.45k/7.45k [00:00<00:00, 35.2MB/s]
config.json: 100%|█████████████████████████████████████████████████████████████████████| 738/738 [00:00<00:00, 3.70MB/s]
model.safetensors.index.json: 100%|████████████████████████████████████████████████| 29.4k/29.4k [00:00<00:00, 23.5MB/s]
.gitattributes: 100%|██████████████████████████████████████████████████████████████| 1.65k/1.65k [00:00<00:00, 6.02MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████| 587/587 [00:00<00:00, 3.50MB/s]
tokenizer_config.json: 100%|███████████████████████████████████████████████████████| 9.35k/9.35k [00:00<00:00, 30.7MB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████| 2.12M/2.12M [00:00<00:00, 2.94MB/s]
model-00002-of-00002.safetensors: 100%|██████████████████████████████████████████████| 610M/610M [01:07<00:00, 9.10MB/s]
stable-code-3b-q4_k_m.gguf: 100%|██████████████████████████████████████████████████| 1.71G/1.71G [01:16<00:00, 22.4MB/s]
stable-code-3b-q5_k_m.gguf: 100%|██████████████████████████████████████████████████| 1.99G/1.99G [02:10<00:00, 15.3MB/s]
model-00001-of-00002.safetensors: 100%|████████████████████████████████████████████| 4.98G/4.98G [02:52<00:00, 28.9MB/s]
Fetching 13 files: 100%|████████████████████████████████████████████████████████████████| 13/13 [02:52<00:00, 13.31s/it]
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux█████████████████| 4.98G/4.98G [02:52<00:00, 58.2MB/s]
Type "help", "copyright", "credits" or "license" for more information.█████████████| 1.99G/1.99G [02:10<00:00, 9.60MB/s]
>>>

example index.html

index.html:22 WebSocket connection to 'ws://localhost:7861/' failed: 
(anonymous) @ index.html:22
index.html:28 WebSocket is already in CLOSING or CLOSED state.
sendMessage @ index.html:28

others

python3 webui.py 
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain/__init__.py:29: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/huggingface_hub/file_download.py:1194: UserWarning: `local_dir_use_symlinks` parameter is deprecated and will be ignored. The process to download files to a local folder has been updated and do not rely on symlinks anymore. You only need to pass a destination folder as`local_dir`.
For more details, check out https://huggingface.co/docs/huggingface_hub/main/en/guides/download#download-files-to-local-folder.
  warnings.warn(
Fetching 13 files: 100%|█████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 11962.69it/s]
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
>>> exit()
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/gradio/components/dropdown.py:93: UserWarning: The `max_choices` parameter is ignored when `multiselect` is False.
  warnings.warn(
/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/gradio/components/dropdown.py:179: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: stabilityai/stable-code-instruct-3b or set allow_custom_value=True.
  warnings.warn(
Traceback (most recent call last):
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain_community/llms/llamacpp.py", line 140, in validate_environment
    from llama_cpp import Llama, LlamaGrammar
ImportError: cannot import name 'LlamaGrammar' from 'llama_cpp' (/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/llama_cpp/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ivo/Projects/Python/LLMinator/LLMinator/webui.py", line 202, in <module>
    llm_chain, llm = init_llm_chain(model_path)
  File "/home/ivo/Projects/Python/LLMinator/LLMinator/webui.py", line 48, in init_llm_chain
    llm = LlamaCpp(
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/pydantic/v1/main.py", line 339, in __init__
    values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/pydantic/v1/main.py", line 1100, in validate_model
    values = validator(cls_, values)
  File "/home/ivo/Projects/Python/LLMinator/lib/python3.10/site-packages/langchain_community/llms/llamacpp.py", line 142, in validate_environment
    raise ImportError(
ImportError: Could not import llama-cpp-python library. Please install the llama-cpp-python library to use this embedding model: pip install llama-cpp-python
IMPORTANT: You are using gradio version 4.27.0, however version 4.29.0 is available, please upgrade.
--------

running python webui.py gets me into console

Hi, I am trying to install and run this project, but after installing requirements and running "make", I am dropped into the console :

instead of running the server. I get no errors, everything installs smoothly (on a new cloud machine, in a fresh conda env with py=3.11).

Am I missing something? Thank you.

Initial execution stops at quantize_model

Hey,

I went by the readme and did the install (into a venv) and a make, but upon starting the webui the execution stops and returns a python interpreter prompt without getting to gradio or loading the model into memory.

I traced the stop point to this piece of code (quantizing of the default model):

llama.cpp was installed fine and I can run the convert-hf-to-gguf.py manually, so I'm a bit stumped as what might be the issue.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.