LLM for Unity enables seamless integration of Large Language Models (LLMs) within the Unity engine.
It allows to create intelligent characters that your players can interact with for an immersive experience.
LLM for Unity is built on top of the awesome llama.cpp and llamafile libraries.
- 💻 Cross-platform! Windows, Linux, macOS and Android
- 🏠 Runs locally without internet access. No data ever leaves the game!
- ⚡ Blazing fast inference on CPU and GPU (Nvidia, AMD, Apple Metal)
- 🤗 Supports all major LLM models
- 🔧 Easy to setup, call with a single line of code
- 💰 Free to use for both personal and commercial purposes
🧪 Tested on Unity: 2021 LTS, 2022 LTS, 2023
🚦 Upcoming Releases
- ⭐ Star the repo, leave us a review and spread the word about the project!
- Join us at Discord and say hi!
- Contribute by submitting feature requests or bugs as issues or even submiting a PR and become a collaborator!
- Verbal Verdict
- I, Chatbot: AISYLUM
- Nameless Souls of the Void
- Murder in Aisle 4
- Finicky Food Delivery AI
- AI Emotional Girlfriend
Method 1: Install using the asset store
- Open the LLM for Unity asset page and click
Add to My Assets
- Open the Package Manager in Unity:
Window > Package Manager
- Select the
Packages: My Assets
option from the drop-down - Select the
LLM for Unity
package, clickDownload
and thenImport
Method 2: Install using the GitHub repo:
- Open the Package Manager in Unity:
Window > Package Manager
- Click the
+
button and selectAdd package from git URL
- Use the repository URL
https://github.com/undreamai/LLMUnity.git
and clickAdd
First you will setup the LLM for your game 🏎:
- Create an empty GameObject.
In the GameObject Inspector clickAdd Component
and select the LLM script. - Download one of the default models with the
Download Model
button (~GBs).
Or load your own .gguf model with theLoad model
button (see LLM model management).
Then you can setup each of your characters as follows 🙋♀️:
- Create an empty GameObject for the character.
In the GameObject Inspector clickAdd Component
and select the LLMCharacter script. - Define the role of your AI in the
Prompt
. You can define the name of the AI (AI Name
) and the player (Player Name
). - (Optional) Select the LLM constructed above in the
LLM
field if you have more than one LLM GameObjects.
You can also adjust the LLM and character settings according to your preference (see Options).
In your script you can then use it as follows 🦄:
using LLMUnity;
public class MyScript {
public LLMCharacter llmCharacter;
void HandleReply(string reply){
// do something with the reply from the model
Debug.Log(reply);
}
void Game(){
// your game function
...
string message = "Hello bot!";
_ = llmCharacter.Chat(message, HandleReply);
...
}
}
You can also specify a function to call when the model reply has been completed.
This is useful if the Stream
option is enabled for continuous output from the model (default behaviour):
void ReplyCompleted(){
// do something when the reply from the model is complete
Debug.Log("The AI replied");
}
void Game(){
// your game function
...
string message = "Hello bot!";
_ = llmCharacter.Chat(message, HandleReply, ReplyCompleted);
...
}
To stop the chat without waiting for its completion you can use:
llmCharacter.CancelRequests();
- Finally, in the Inspector of the GameObject of your script, select the LLMCharacter GameObject created above as the llmCharacter property.
That's all ✨!
You can also:
Build a mobile app on Android
To build an Android app you need to specify the IL2CPP
scripting backend and the ARM64
as the target architecture in the player settings.
These settings can be accessed from the Edit > Project Settings
menu within the Player > Other Settings
section.
![](.github/android.png)
It is also a good idea to enable the Download on Build
option in the LLM GameObject to download the model on launch in order to keep the app size small.
Save / Load your chat history
To automatically save / load your chat history, you can specify the Save
parameter of the LLMCharacter to the filename (or relative path) of your choice.
The file is saved in the persistentDataPath folder of Unity.
This also saves the state of the LLM which means that the previously cached prompt does not need to be recomputed.
To manually save your chat history, you can use:
llmCharacter.Save("filename");
and to load the history:
llmCharacter.Load("filename");
where filename the filename or relative path of your choice.
Process the prompt at the beginning of your app for faster initial processing time
void WarmupCompleted(){
// do something when the warmup is complete
Debug.Log("The AI is nice and ready");
}
void Game(){
// your game function
...
_ = llmCharacter.Warmup(WarmupCompleted);
...
}
Decide whether or not to add the message to the chat/prompt history
The last argument of the Chat
function is a boolean that specifies whether to add the message to the history (default: true):
void Game(){
// your game function
...
string message = "Hello bot!";
_ = llmCharacter.Chat(message, HandleReply, ReplyCompleted, false);
...
}
Use pure text completion
void Game(){
// your game function
...
string message = "The cat is away";
_ = llmCharacter.Complete(message, HandleReply, ReplyCompleted);
...
}
Wait for the reply before proceeding to the next lines of code
For this you can use the async
/await
functionality:
async void Game(){
// your game function
...
string message = "Hello bot!";
string reply = await llmCharacter.Chat(message, HandleReply, ReplyCompleted);
Debug.Log(reply);
...
}
Add a LLM / LLMCharacter component programmatically
using UnityEngine;
using LLMUnity;
public class MyScript : MonoBehaviour
{
LLM llm;
LLMCharacter llmCharacter;
async void Start()
{
// disable gameObject so that theAwake is not called immediately
gameObject.SetActive(false);
// Add an LLM object
llm = gameObject.AddComponent<LLM>();
// set the model using the filename of the model.
// The model needs to be added to the LLM model manager (see LLM model management) by loading or downloading it.
// Otherwise the model file can be copied directly inside the StreamingAssets folder.
llm.SetModel("Phi-3-mini-4k-instruct-q4.gguf");
// optional: you can also set a lora in a similar fashion
llm.SetLora("my-lora.bin");
// optional: you can set the chat template of the model if it is not correctly identified
// You can find a list of chat templates in the ChatTemplate.templates.Keys
llm.SetTemplate("phi-3");
// optional: set number of threads
llm.numThreads = -1;
// optional: enable GPU by setting the number of model layers to offload to it
llm.numGPULayers = 10;
// Add an LLMCharacter object
llmCharacter = gameObject.AddComponent<LLMCharacter>();
// set the LLM object that handles the model
llmCharacter.llm = llm;
// set the character prompt
llmCharacter.SetPrompt("A chat between a curious human and an artificial intelligence assistant.");
// set the AI and player name
llmCharacter.AIName = "AI";
llmCharacter.playerName = "Human";
// optional: set streaming to false to get the complete result in one go
// llmCharacter.stream = true;
// optional: set a save path
// llmCharacter.save = "AICharacter1";
// optional: enable the save cache to avoid recomputation when loading a save file (requires ~100 MB)
// llmCharacter.saveCache = true;
// optional: set a grammar
// await llmCharacter.SetGrammar("json.gbnf");
// re-enable gameObject
gameObject.SetActive(true);
}
}
Use a remote server
You can use a remote server to carry out the processing and implement characters that interact with it. To do that:
- Create a project with a GameObject using the
LLM
script as described above. Enable theRemote
option and optionally configure the port. - Create a second project with the game characters using the
LLMCharacter
script as described above. Enable theRemote
option and configure the host with the IP address (starting with "http://") and port of the server.
A detailed documentation on function level can be found here:
The Samples~ folder contains several examples of interaction 🤖:
- SimpleInteraction: Demonstrates a simple interaction with an AI character
- MultipleCharacters: Demonstrates a simple interaction using multiple AI characters
- KnowledgeBaseGame: Simple detective game using a knowledge base to provide information to the LLM based on google/mysteryofthreebots
- ChatBot: Demonstrates interaction between a player and a AI with a UI similar to a messaging app (see image below)
- AndroidDemo: Example Android app with an initial screen with model download progress
To install a sample:
- Open the Package Manager:
Window > Package Manager
- Select the
LLM for Unity
Package. From theSamples
Tab, clickImport
next to the sample you want to install.
The samples can be run with the Scene.unity
scene they contain inside their folder.
In the scene, select the LLM
GameObject and click the Download Model
button to download a default model or Load model
to load your own model (see LLM model management).
Save the scene, run and enjoy!
LLM for Unity implements a model manager that allows to load or download LLMs and ship them directly in your game.
The model manager can be found as part of the LLM GameObject:
You can download models with the Download model
button.
LLM for Unity includes different state of the art models built-in for different model sizes, quantised with the Q4_K_M method.
Alternative models can be downloaded from HuggingFace in the .gguf format.
You can download a model locally and load it with the Load model
button, or copy the URL in the Download model > Custom URL
field to directly download it.
If a HuggingFace model does not provide a gguf file, it can be converted to gguf with this online converter.
The chat template used for constructing the prompts is determined automatically from the model (if a relevant entry exists) or the model name.
If incorrecly identified, you can select another template from the chat template dropdown.
Models added in the model manager are copied to the game during the building process.
You can omit a model from being built in by deselecting the "Build" checkbox.
To remove the model (but not delete it from disk) you can click the bin button.
The the path and URL (if downloaded) of each added model is diplayed in the expanded view of the model manager access with the >>
button:
You can create lighter builds by selecting the Download on Build
option.
Using this option the models will be downloaded the first time the game starts instead of copied in the build.
If you have loaded a model locally you need to set its URL through the expanded view, otherwise it will be copied in the build.
❕ Before using any model make sure you check their license ❕
Show/Hide Advanced Options
Toggle to show/hide advanced options from belowLog Level
select how verbose the log messages are
-
Remote
select to provide remote access to the LLM -
Port
port to run the LLM server (ifRemote
is set) -
Num Threads
number of threads to use (default: -1 = all) -
Num GPU Layers
number of model layers to offload to the GPU. If set to 0 the GPU is not used. Use a large number i.e. >30 to utilise the GPU as much as possible. Note that higher values of context size will use more VRAM. If the user's GPU is not supported, the LLM will fall back to the CPU -
Debug
select to log the output of the model in the Unity Editor -
Advanced options
Parallel Prompts
number of prompts that can happen in parallel (default: -1 = number of LLMCharacter objects)Dont Destroy On Load
select to not destroy the LLM GameObject when loading a new Scene
-
Download model
click to download one of the default models -
Load model
click to load your own model in .gguf format -
Download on Start
enable to downloaded the LLM models the first time the game starts. Alternatively the LLM models wil be copied directly in the build -
Advanced options
Download lora
click to download a LoRA model in .bin formatLoad lora
click to load a LoRA model in .bin format-
This is the number of tokens the model can take as input when generating responses. Higher values use more RAM or VRAM (if using GPU).Context Size
size of the prompt context (0 = context size of the model) Batch Size
batch size for prompt processing (default: 512)Model
the path of the model being used (relative to the Assets/StreamingAssets folder)Chat Template
the chat template being used for the LLMLora
the path of the LoRA being used (relative to the Assets/StreamingAssets folder)
-
Advanced options
-
Base Prompt
a common base prompt to use across all LLMCharacter objects using the LLM
Show/Hide Advanced Options
Toggle to show/hide advanced options from belowLog Level
select how verbose the log messages are
Remote
whether the LLM used is remote or localLLM
the LLM GameObject (ifRemote
is not set)Hort
ip of the LLM (ifRemote
is set)Port
port of the LLM (ifRemote
is set)-
If set, the chat history and LLM state (if save cache is enabled) is automatically saved to file specified.Save
save filename or relative path
The chat history is saved with a json suffix and the LLM state with a cache suffix.
Both files are saved in the [persistentDataPath folder of Unity](https://docs.unity3d.com/ScriptReference/Application-persistentDataPath.html). Save Cache
select to save the LLM state along with the chat history. The LLM state is typically around 100MB+.Debug Prompt
select to log the constructed prompts in the Unity Editor
Player Name
the name of the playerAI Name
the name of the AIPrompt
description of the AI role
-
Stream
select to receive the reply from the model as it is produced (recommended!).
If it is not selected, the full reply from the model is received in one go -
Advanced options
Load grammar
click to load a grammar in .gbnf formatGrammar
the path of the grammar being used (relative to the Assets/StreamingAssets folder)-
Saves the prompt while it is being created by the chat to avoid reprocessing the entire prompt every timeCache Prompt
save the ongoing prompt from the chat (default: true) Seed
seed for reproducibility. For random results every time use -1-
This is the maximum amount of tokens the model will maximum predict. When N tokens are reached the model will stop generating. This means words / sentences might not get finished if this is too low.Num Predict
maximum number of tokens to predict (default: 256, -1 = infinity, -2 = until context filled) -
The temperature setting adjusts how random the generated responses are. Turning it up makes the generated choices more varied and unpredictable. Turning it down makes the generated responses more predictable and focused on the most likely options.Temperature
LLM temperature, lower values give more deterministic answers (default: 0.2) -
The top k value controls the top k most probable tokens at each step of generation. This value can help fine tune the output and make this adhere to specific patterns or constraints.Top K
top-k sampling (default: 40, 0 = disabled) -
The top p value controls the cumulative probability of generated tokens. The model will generate tokens until this theshold (p) is reached. By lowering this value you can shorten output & encourage / discourage more diverse outputs.Top P
top-p sampling (default: 0.9, 1.0 = disabled) -
The probability is defined relative to the probability of the most likely token.Min P
minimum probability for a token to be used (default: 0.05) -
The penalty is applied to repeated tokens.Repeat Penalty
control the repetition of token sequences in the generated text (default: 1.1) -
Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.Presence Penalty
repeated token presence penalty (default: 0.0, 0.0 = disabled) -
Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.Frequency Penalty
repeated token frequency penalty (default: 0.0, 0.0 = disabled) Tfs_z
: enable tail free sampling with parameter z (default: 1.0, 1.0 = disabled).Typical P
: enable locally typical sampling with parameter p (default: 1.0, 1.0 = disabled).Repeat Last N
: last N tokens to consider for penalizing repetition (default: 64, 0 = disabled, -1 = ctx-size).Penalize Nl
: penalize newline tokens when applying the repeat penalty (default: true).Penalty Prompt
: prompt for the purpose of the penalty evaluation. Can be eithernull
, a string or an array of numbers representing tokens (default:null
= use originalprompt
).Mirostat
: enable Mirostat sampling, controlling perplexity during text generation (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0).Mirostat Tau
: set the Mirostat target entropy, parameter tau (default: 5.0).Mirostat Eta
: set the Mirostat learning rate, parameter eta (default: 0.1).N Probs
: if greater than 0, the response also contains the probabilities of top N tokens for each generated token (default: 0)Ignore Eos
: enable to ignore end of stream tokens and continue generating (default: false).
The license of LLM for Unity is MIT (LICENSE.md) and uses third-party software with MIT and Apache licenses (Third Party Notices.md).