Topic: llm-serving Goto Github

👇 Here are 45 public repositories matching this topic...

alibaba / rtp-llm
llm-serving,RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
Organization: alibaba
gpt inference llama llm llm-serving llmops model-serving

antoniogr7 / pratical-llms
llm-serving,A collection of hand on notebook for LLMs practitioner
User: antoniogr7
genai llm llm-evaluation llm-inference llm-serving llm-training quantization

asprenger / ray_vllm_inference
llm-serving,A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
User: asprenger
inference llm llm-serving llmops mlops model-serving pytorch ray transformer vllm

azminewasi / awesome-llms-iclr-24
llm-serving,It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on Learning Representations (ICLR) in 2024.
User: azminewasi
large-language-model large-language-models large-language-models-and-translation-systems large-language-models-for-graph-learning llm llm-agent llm-evaluation llm-framework llm-inference llm-privacy

bentoml / openllm
llm-serving,Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.
Organization: bentoml
Home Page: https://bentoml.com
llm llmops model-inference falcon fine-tuning stablelm llm-serving llama mpt vicuna

biosfood / intel-llm-guide
llm-serving,A guide on how to run LLMs on intel CPUs
User: biosfood
guide intel llm llm-inference llm-serving machine-learning setup setup-development-environment tutorial

centml / llm-inference-bench
llm-serving,Lightweight and extensible LLM Inference serving benchmark tool written in Rust.
Organization: centml
benchmarking llm-inference llm-serving

chenhunghan / ialacol
llm-serving,🪶 Lightweight OpenAI drop-in replacement for Kubernetes
User: chenhunghan
ai helm kubernetes langchain llm python openai cloudnative ggml gpu

ehsanghaffar / ein-llm
llm-serving,A self-hosted personal chatbot API with FastAPI. It allows you to interact with the Llama2 LLM (and other open-source LLMs) to have natural language conversations, generate text, and perform various language-related tasks.
User: ehsanghaffar
langchain llama2 llamacpp llm-serving

fork123aniket / llm-rag-powered-qa-app
llm-serving,A Production-Ready, Scalable RAG-powered LLM-based Context-Aware QA App
User: fork123aniket
context-aware-system eleutherai fine-tuning large-language-models llm-inference llm-serving llm-training llmops parameter-efficient-fine-tuning question-answering

friendliai / friendli-client
llm-serving,Friendli: the fastest serving engine for generative AI
Organization: friendliai
Home Page: https://friendli.ai
generative-ai llm llm-inference llmops serving gpt gpt3 inference llama2 llm-serving

galeselee / awesome_llm_system-paperlist
llm-serving,Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inference acceleration, and related works will be gradually added in the future. Welcome contributions!
User: galeselee
Home Page: https://galeselee.gitbook.io/awesome-papers/
llm-inference llm-serving paperlist papers system

george-mountain / llm-local-streaming
llm-serving,Streaming of LLM responses in realtime using Fastapi and Streamlit.
User: george-mountain
ai fastapi llm llm-serving streamlit llm-streaming

george-mountain / web-app-builder--llm
llm-serving,Building Static Web Applications using Large Language Model. From hand sketched documents, images and screenshots to proper web pages.
User: george-mountain
ai llm llm-serving pypi pypi-package streamlit

hpcaitech / swiftinfer
llm-serving,Efficient AI Inference & Serving
Organization: hpcaitech
Home Page: https://hpc-ai.com/
artificial-intelligence deep-learning gpt inference llama llama2 llm-inference llm-serving

hpmll / burstgpt
llm-serving,A GPT-3.5 & GPT-4 Workload Trace to Optimize LLM Serving Systems
Organization: hpmll
dataset llm mlsys llm-serving

inquestgeronimo / horizon-takeoff
llm-serving,Automating the deployment of the Takeoff Server on AWS for LLMs
User: inquestgeronimo
artificial-intelligence aws cloud deep-learning ec2 llm-inference llm-serving llmops machine-learning server

ivanlulyf / bunny-llm
llm-serving,Deno LLM API Service
User: ivanlulyf
Home Page: https://bunny-gpt.deno.dev/
chatgpt chatgpt-api cloudflare-workers-ai llm llm-serving

ivynya / illm
llm-serving,internet llm - access your ollama (or any other local llm) instance from across the internet
User: ivynya
llm-serving ollama ollama-interface

liguodongiot / llm-action
llm-serving,本项目旨在分享大模型相关技术原理以及实战经验。
User: liguodongiot
Home Page: https://www.zhihu.com/column/c_1456193767213043713
llm llm-inference llm-serving llm-training llmops

liux2 / dl_env_setups
llm-serving,Deep learning environment setups
User: liux2
deep-learning docker environments llm-serving

loopglitch26 / hinglish-ai-mentor
llm-serving,Hinglish Chatbot powered by Azure Cognitive Services, Google Translate and Open AI
User: loopglitch26
Home Page: https://hinglish-chatbot-loopglitch26.streamlit.app/
azure google llm-serving nlp-machine-learning prompt-engineering

mani-kantap / llm-inference-solutions
llm-serving,A collection of all available inference solutions for the LLMs
User: mani-kantap
llm-inference llm-serving llmops

mddunlap924 / llm-inference-serving
llm-serving,This repository demonstrates LLM execution on CPUs using packages like llamafile, emphasizing low-latency, high-throughput, and cost-effective benefits for inference and serving.
User: mddunlap924
deepspeed large-language-models llamacpp llamafile llm-inference llm-serving llms vllm

microsoft / aici
llm-serving,AICI: Prompts as (Wasm) Programs
Organization: microsoft
ai rust wasm wasmtime inference language-model llm llm-framework llm-inference llm-serving llmops model-serving transformer

mosecorg / mosec
llm-serving,A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
Organization: mosecorg
Home Page: https://mosec.readthedocs.io/
model-serving deep-learning machine-learning nerual-network mlops machine-learning-platform hacktoberfest gpu python pytorch

oscinis-com / awesome-llm-productization
llm-serving,Awesome-LLM-Productization: a curated list of tools/tricks/news/regulations about AI and Large Language Model (LLM) productization
Organization: oscinis-com
ai large-language-models llm llm-inference llm-serving llmops machine-learning mlops production transformer

oss-pole-emploi / happy_vllm
llm-serving,A REST API for vLLM, production ready
Organization: oss-pole-emploi
Home Page: https://oss-pole-emploi.github.io/happy_vllm/
api-rest llm llm-serving mlops production serving transformers vllm

predibase / lorax
llm-serving,Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
Organization: predibase
Home Page: https://loraexchange.ai
fine-tuning gpt llama llm llm-inference llm-serving llmops lora model-serving pytorch transformers

ray-project / anyscale-berkeley-ai-hackathon
llm-serving,Ray and Anyscale for UC Berkeley AI Hackathon!
Organization: ray-project
Home Page: https://ai.calhacks.io/
anyscale berkeley-ai hackathon llm llm-inference llm-serving ray-distributed

ray-project / llm-application
llm-serving,
Organization: ray-project
anyscale llm llm-serving ray-distributed nlp scalable-machine-learning

ray-project / llms-in-prod-workshop-2023
llm-serving,Deploy and Scale LLM-based applications
Organization: ray-project
Home Page: https://home.mlops.community/public/events/ray-workshop-2023-06-15
anyscale llm llm-inference llm-serving llms ray

ray-project / ray
llm-serving,Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Organization: ray-project
Home Page: https://ray.io
ray distributed parallel machine-learning reinforcement-learning deep-learning python rllib hyperparameter-search optimization

ray-project / ray-educational-materials
llm-serving,This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.
Organization: ray-project
deep-learning distributed-machine-learning ray-distributed ray-tune ray ray-train ray-data ray-serve generative-ai llm

ray-project / ray-llm
llm-serving,RayLLM - LLMs on Ray
Organization: ray-project
Home Page: https://aviary.anyscale.com
distributed-systems large-language-models ray serving transformers llm llm-inference llm-serving llmops

rohan-paul / llm-finetuning-large-language-models
llm-serving,LLM (Large Language Model) FineTuning
User: rohan-paul
gpt-3 gpt3-turbo large-language-models llama2 llm llm-finetuning llm-inference llm-serving llm-training mistral-7b

skypilot-org / skypilot
llm-serving,SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.
Organization: skypilot-org
Home Page: https://skypilot.readthedocs.io
cloud-computing data-science deep-learning gpu hyperparameter-tuning machine-learning tpu job-queue job-scheduler cloud-management

slai-labs / get-beam
llm-serving,Run GPU inference and training jobs on serverless infrastructure that scales with you.
Organization: slai-labs
Home Page: https://beam.cloud
artificial-intelligence machine-learning python serverless cloud-computing cost-optimization data-science deep-learning distributed-computing gpu-acceleration

stosan / commentator
llm-serving,
User: stosan
Home Page: https://code-commentator.onrender.com/
generative-ai llm-serving

substratusai / runbooks
llm-serving,Finetune LLMs on K8s by using Runbooks
Organization: substratusai
Home Page: https://www.substratus.ai
kubernetes mlops llmops kubernetes-operator llm-inference llm-serving llm-training ml-platform

sugarcane-ai / sugarcane-ai
llm-serving,npm like package ecosystem for Prompts 🤖
Organization: sugarcane-ai
Home Page: https://sugarcaneai.dev
ai llm prompt-engineering framework llm-chain llm-framework llm-serving llm-training llmops next

sugarcane-ai / sugarcane-ai.github.io
llm-serving,
Organization: sugarcane-ai
Home Page: https://sugarcaneai.dev
llm llmops generative-ai framework workflow reusable typescript api prompt prompt-engineering

suleymansevimli / run-llm-model-locally
llm-serving,You can run any large language model on your local machine with this repository.
User: suleymansevimli
git-lfs huggingface llm llm-serving python

superduperdb / superduperdb
llm-serving,🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalable model training and vector search.
Organization: superduperdb
Home Page: https://superduperdb.com
ai chatbot data database distributed-ml inference llm-inference llm-serving llmops ml mlops mongodb pretrained-models python pytorch rag semantic-search torch transformers vector-search

vllm-project / vllm
llm-serving,A high-throughput and memory-efficient inference and serving engine for LLMs
Organization: vllm-project
Home Page: https://docs.vllm.ai
gpt llm pytorch llmops mlops model-serving transformer llm-serving inference llama