• llama-node
  • Introduction
  • Installation
  • Manual compilation
  • Acknowledgments
    • Models/Inferencing tools dependencies
    • Some source code comes from

This project is in an early stage, the API for nodejs may change in the future, use it with caution.

This is a nodejs library for inferencing llama, rwkv or llama derived models. It was built on top of llama-rs, llama.cpp and rwkv.cpp. It uses napi-rs for channel messages between node.js and llama thread.

Currently supported models (All of these should be converted to GGML format):

• LLaMA
• RWKV
• Alpaca
• GPT4All
• Chinese LLaMA / Alpaca
• Vigogne (French)
• Vicuna
• Koala

Supported platforms:

• darwin-x64
• darwin-arm64
• linux-x64-gnu (glibc >= 2.31)
• linux-x64-musl
• win32-x64-msvc

Node.js version: >= 16

• Install llama-node npm package

npm install llama-node

• Install anyone of the inference backends (at least one)

  • llama.cpp

  npm install @llama-node/llama-cpp

  • or llama-rs

  npm install @llama-node/core

  • or rwkv.cpp

  npm install @llama-node/rwkv-cpp

WIP, will be published on our official websites soon.

This library was published under MIT/Apache-2.0 license. However, we stronly recommend you to cite our work/our dependencies work if you wish to reuse the code from this library.

• LLaMA models: facebookresearch/llama
• RWKV models: BlinkDL/RWKV-LM
• llama.cpp: ggreganov/llama.cpp
• llama-rs: rustformers/llama-rs
• rwkv.cpp: saharNooby/rwkv.cpp

• cpp-rust bindings build scripts: sobelio/llm-chain
• rwkv logits sampling: KerfuffleV2/smolrsrwkv