LLaMa2 GPTQ

Chat AI which can provide responses with reference documents by Prompt engineering over vector database. It suggests related web pages provided through the integration with my previous product, Texonom.

Pursuing local, private and personal AI without requesting external API attained by optimizing inference performance with GPTQ model quantization. This project was inspired by the langchain projects like notion-qa, localGPT.

Demos

CLI Demo

cli.mp4

Chat Demo

chat.mp4

Install

This project is using rye as package manager Currently only available with CUDA

rye sync

or using pip

CUDA_VERSION=cu118
TORCH_VERSION=2.0.1
pip install torch==$TORCH_VERSION --index-url https://download.pytorch.org/whl/$CUDA_VERSION --force
pip install torch==$TORCH_VERSION --index-url https://download.pytorch.org/whl/$CUDA_VERSION
pip install .

QA

1. Chat with Web UI

streamlit run chat.py

2. Chat with CLI

python main.py chat

Ingest Documents

Currently code structure is mainly focussed on Notion's csv exported data

Custom source documents

# Put document files to ./knowledge folder
python main.py process
# Or use provided Texonom DB
git clone https://huggingface.co/datasets/texonom/md-chroma-instructor-xl db

Quantize Model

Default model is orca 3b for now

python main quantize --source_model facebook/opt-125m --output opt-125m-4bit-gptq --push

Future Plan

MPS support using dynamic model selecting
Stateful Web App support like chat-langchain

App Stack

LLM Stack

Langchain for Prompt Engineering
ChromaDB for storing embeddings
Transformers for LLM engine
AutoGPTQ for Quantization & Inference

Python Stack

Rye for package management
Mypy for type checking
Fire for CLI implementation
Streamlit for Web UI implementation

deps: renovate dependency dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

deps: Update all non-major dependencies (accelerate, aiohttp, argilla, async-timeout, auto-gptq, auto_gptq, certifi, chardet, charset-normalizer, chromadb, click, clickhouse-connect, cryptography, dataclasses-json, datasets, dill, fastapi, frozenlist, gitpython, httpcore, httptools, httpx, huggingface-hub, importlib-metadata, joblib, jsonschema, jsonschema-specifications, langchain, markdown, marshmallow, multiprocess, numexpr, numpy, overrides, protobuf, pydantic, pygments, python, pyyaml, referencing, regex, rich, rpds-py, safetensors, sqlalchemy, starlette, streamlit, threadpoolctl, tqdm, transformers, typer, unstructured, urllib3, uvicorn, validators, wrapt, xxhash, zipp)

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

pep621

pyproject.toml

langchain ~=0.0.225

chromadb ~=0.3.26

transformers ~=4.30.2

InstructorEmbedding ~=1.0.1

sentence_transformers ~=2.2.2

unstructured ~=0.7.12

torch >=2.0.1

auto_gptq ~=0.2.2

einops ~=0.6.1

fire ~=0.5.0

streamlit-chat ~=0.1.1

protobuf <=3.20.0

pip_requirements

requirements.lock

accelerate ==0.20.3

aiohttp ==3.8.4

aiosignal ==1.3.1

altair ==5.0.1

anyio ==3.7.1

argilla ==1.12.0

async-timeout ==4.0.2

attrs ==23.1.0

auto-gptq ==0.2.2

backoff ==2.2.1

blinker ==1.6.2

cachetools ==5.3.1

certifi ==2023.5.7

cffi ==1.15.1

chardet ==5.1.0

charset-normalizer ==3.1.0

chromadb ==0.3.26

click ==8.1.3

clickhouse-connect ==0.6.5

colorama ==0.4.6

coloredlogs ==15.0.1

commonmark ==0.9.1

cryptography ==41.0.1

dataclasses-json ==0.5.9

datasets ==2.13.1

decorator ==5.1.1

deprecated ==1.2.14

dill ==0.3.6

duckdb ==0.8.1

einops ==0.6.1

et-xmlfile ==1.1.0

exceptiongroup ==1.1.2

fastapi ==0.99.1

filelock ==3.12.2

filetype ==1.2.0

fire ==0.5.0

flatbuffers ==23.5.26

frozenlist ==1.3.3

fsspec ==2023.6.0

gitdb ==4.0.10

gitpython ==3.1.31

greenlet ==2.0.2

h11 ==0.14.0

hnswlib ==0.7.0

httpcore ==0.16.3

httptools ==0.5.0

httpx ==0.23.3

huggingface-hub ==0.16.2

humanfriendly ==10.0

idna ==3.4

importlib-metadata ==6.7.0

instructorembedding ==1.0.1

jinja2 ==3.1.2

joblib ==1.3.1

jsonschema ==4.18.0

jsonschema-specifications ==2023.6.1

langchain ==0.0.225

langchainplus-sdk ==0.0.20

lxml ==4.9.3

lz4 ==4.3.2

markdown ==3.4.3

markupsafe ==2.1.3

marshmallow ==3.19.0

marshmallow-enum ==1.5.1

monotonic ==1.6

mpmath ==1.3.0

msg-parser ==1.2.0

multidict ==6.0.4

multiprocess ==0.70.14

mypy-extensions ==1.0.0

networkx ==3.1

nltk ==3.8.1

numexpr ==2.8.4

numpy ==1.23.5

olefile ==0.46

onnxruntime ==1.15.1

openapi-schema-pydantic ==1.2.4

openpyxl ==3.1.2

overrides ==7.3.1

packaging ==23.1

pandas ==1.5.3

pdf2image ==1.16.3

pdfminer-six ==20221105

pillow ==9.5.0

posthog ==3.0.1

protobuf ==3.20.0

psutil ==5.9.5

pulsar-client ==3.2.0

pyarrow ==12.0.1

pycparser ==2.21

pydantic ==1.10.11

pydeck ==0.8.1b0

pygments ==2.15.1

pympler ==1.0.1

pypandoc ==1.11

pyreadline3 ==3.4.1

python-dateutil ==2.8.2

python-docx ==0.8.11

python-dotenv ==1.0.0

python-magic ==0.4.27

python-pptx ==0.6.21

pytz ==2023.3

pytz-deprecation-shim ==0.1.0.post0

pyyaml ==6.0

referencing ==0.29.1

regex ==2023.6.3

requests ==2.31.0

rfc3986 ==1.5.0

rich ==13.0.1

rouge ==1.0.1

rpds-py ==0.8.7

safetensors ==0.3.1

scikit-learn ==1.3.0

scipy ==1.11.1

sentence-transformers ==2.2.2

sentencepiece ==0.1.99

six ==1.16.0

smmap ==5.0.0

sniffio ==1.3.0

sqlalchemy ==2.0.18

starlette ==0.27.0

streamlit ==1.24.0

streamlit-chat ==0.1.1

sympy ==1.12

tabulate ==0.9.0

tenacity ==8.2.2

termcolor ==2.3.0

threadpoolctl ==3.1.0

tokenizers ==0.13.3

toml ==0.10.2

toolz ==0.12.0

torch ==2.0.1+cu118

torchvision ==0.15.2+cu118

tornado ==6.3.2

tqdm ==4.65.0

transformers ==4.30.2

typer ==0.7.0

typing-extensions ==4.7.1

typing-inspect ==0.9.0

tzdata ==2023.3

tzlocal ==4.3.1

unstructured ==0.7.12

urllib3 ==2.0.3

uvicorn ==0.22.0

validators ==0.20.0

watchdog ==3.0.0

watchfiles ==0.19.0

websockets ==11.0.3

wrapt ==1.14.1

xlrd ==2.0.1

xlsxwriter ==3.1.2

xxhash ==3.2.0

yarl ==1.9.2

zipp ==3.15.0

zstandard ==0.21.0

pyenv

.python-version

python 3.10.0

Check this box to trigger a request for Renovate to run again on this repository

seonglae / llama2gptq Goto Github PK

llama2gptq's Introduction

LLaMa2 GPTQ

Demos

CLI Demo

Chat Demo

Install

QA

1. Chat with Web UI

2. Chat with CLI

Ingest Documents

Custom source documents

Quantize Model

Future Plan

App Stack

LLM Stack

Python Stack

llama2gptq's People

Contributors

Stargazers

Watchers

llama2gptq's Issues

Open

Ignored or Blocked

Detected dependencies

Recommend Projects

Recommend Topics

Recommend Org