This project is in an early stage, the API for nodejs may change in the future, use it with caution.
This is a nodejs library for inferencing llama, rwkv or llama derived models. It was built on top of llama-rs, llama.cpp and rwkv.cpp. It uses napi-rs for channel messages between node.js and llama thread.
Currently supported models (All of these should be converted to GGML format):
Supported platforms:
- darwin-x64
- darwin-arm64
- linux-x64-gnu (glibc >= 2.31)
- linux-x64-musl
- win32-x64-msvc
Node.js version: >= 16
- Install llama-node npm package
npm install llama-node
-
Install anyone of the inference backends (at least one)
- llama.cpp
npm install @llama-node/llama-cpp
- or llama-rs
npm install @llama-node/core
- or rwkv.cpp
npm install @llama-node/rwkv-cpp
WIP, will be published on our official websites soon.
This library was published under MIT/Apache-2.0 license. However, we stronly recommend you to cite our work/our dependencies work if you wish to reuse the code from this library.
- LLaMA models: facebookresearch/llama
- RWKV models: BlinkDL/RWKV-LM
- llama.cpp: ggreganov/llama.cpp
- llama-rs: rustformers/llama-rs
- rwkv.cpp: saharNooby/rwkv.cpp
- cpp-rust bindings build scripts: sobelio/llm-chain
- rwkv logits sampling: KerfuffleV2/smolrsrwkv