Giter VIP home page Giter VIP logo

llm_load_balancer's Introduction

CSL2090 Course Project:

Team Members

  • Jyotin Goel (B22AI063)

Installation instructions for CLI tool

tar zxvf CSL2090_PCS2_CourseProject_ollama_Linux_x86_64.tar.gz
sudo sh install.sh

Demo

Demo

Instructions to use CLI tool

jpt setserver <host>:<port> <model>

Load Balancer Setup Instructions

Step 1: Start the ollama instances

Here we take a sample of 3

docker run -d -v ollama:/root/.ollama -p 3030:11434 --name ollama1 ollama/ollama
docker run -d -v ollama:/root/.ollama -p 11432:11434 --name ollama2 ollama/ollama
docker run -d -v ollama:/root/.ollama -p 11433:11434 --name ollama3 ollama/ollama

(Optional)

docker exec -it <container_id> bash
#/ ollama pull gemma:2b

Step 2: Build the nginx image

cd nginx_app/
docker build -t loadbalancer .

Step 3: Run the nginx image

docker run -p 8080:8080 --network="host" loadbalancer

Concepts used:

  • Docker Container

  • Least number of active connections load balancing

  • Round Robin Load balancing

Step 4: Run the jpt client:

jpt setserver 127.0.0.1:8080 gemma:2b

In case you want to access another person's hosted llms

jpt setserver <hosted_ip>:8080 gemma:2b

llm_load_balancer's People

Contributors

gjyotin305 avatar

Stargazers

 avatar Saahil Bhavsar avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.