⚗️ Instill Model manages the AI model-related resources and features working with Instill VDP.
☁️ Instill Cloud offers fully managed Instill Model. Please sign up to try out for free.
-
⚡️ High-performing inference implemented in Go with Triton Inference Server for unleashing the full power of NVIDIA GPU architecture (e.g., concurrency, scheduler, batcher) supporting TensorRT, PyTorch, TensorFlow, ONNX, Python and more.
-
🖱️ One-click model deployment from GitHub, Hugging Face or cloud storage managed by version control tools like DVC or ArtiVC.
-
📦 Standardised AI Task output formats to streamline data integration or analysis
-
macOS or Linux - VDP works on macOS or Linux, but does not support Windows yet.
-
Docker and Docker Compose - VDP uses Docker Compose (specifically,
Compose V2
andCompose specification
) to run all services at local. Please install the latest stable Docker and Docker Compose before using VDP. -
yq
>v4.x
. Please follow the installation guide. -
(Optional) NVIDIA Container Toolkit - To enable GPU support in VDP, please refer to NVIDIA Cloud Native Documentation to install NVIDIA Container Toolkit. If you'd like to specifically allot GPUs to VDP, you can set the environment variable
NVIDIA_VISIBLE_DEVICES
. For example,NVIDIA_VISIBLE_DEVICES=0,1
will make thetriton-server
consume GPU device id0
and1
specifically. By defaultNVIDIA_VISIBLE_DEVICES
is set toall
to use all available GPUs on the machine.
Execute the following commands to start pre-built images with all the dependencies:
The stable release version
$ git clone -b v0.4.0-alpha https://github.com/instill-ai/model.git && cd model
# Launch all services
$ make all
The latest version for development
$ git clone https://github.com/instill-ai/model.git && cd model
# Launch all services
$ make latest PROFILE=all
Note Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.
Note The image of model-backend (~2GB) and Triton Inference Server (~23GB) can take a while to pull, but this should be an one-time effort at the first setup.
We curate a list of ready-to-use models. These pre-trained models are from different sources and have been trained and deployed by our team. Want to contribute a new model? Please create an issue, we are happy to add it to the list 👐.
Model | Task | Sources | Framework | CPU | GPU |
---|---|---|---|---|---|
MobileNet v2 | Image Classification | GitHub-DVC | ONNX | ✅ | ✅ |
Vision Transformer (ViT) | Image Classification | Hugging Face | ONNX | ✅ | ❌ |
YOLOv4 | Object Detection | GitHub-DVC | ONNX | ✅ | ✅ |
YOLOv7 | Object Detection | GitHub-DVC | ONNX | ✅ | ✅ |
YOLOv7 W6 Pose | Keypoint Detection | GitHub-DVC | ONNX | ✅ | ✅ |
PSNet + EasyOCR | Optical Character Recognition (OCR) | GitHub-DVC | ONNX | ✅ | ✅ |
Mask RCNN | Instance Segmentation | GitHub-DVC | PyTorch | ✅ | ✅ |
Lite R-ASPP based on MobileNetV3 | Semantic Segmentation | GitHub-DVC | ONNX | ✅ | ✅ |
Stable Diffusion | Text to Image | GitHub-DVC, Local-CPU, Local-GPU | ONNX | ✅ | ✅ |
Megatron GPT2 | Text Generation | GitHub-DVC | FasterTransformer | ❌ | ✅ |
Llama2 | Text Generation | GitHub-DVC | vLLM, PyTorch | ✅ | ✅ |
Llama2 Chat | Text Generation | GitHub-DVC | vLLM | ❌ | ✅ |
Code Llama | Text Generation | GitHub-DVC | vLLM | ❌ | ✅ |
MosaicML Mpt | Text Generation | GitHub-DVC | vLLM | ❌ | ✅ |
Mistral | Text Generation | GitHub-DVC | vLLM | ❌ | ✅ |
Llava | Text Generation | GitHub-DVC | PyTorch | ❌ | ✅ |
Note: The GitHub-DVC
source in the table means importing a model into VDP from a GitHub repository that uses DVC to manage large files.
📔 Documentation
Please check out the documentation website.
📘 API Reference
The gRPC protocols in protobufs provide the single source of truth for the VDP APIs. The genuine protobuf documentation can be found in our Buf Scheme Registry (BSR).
For the OpenAPI documentation, access http://localhost:3001 after make all
, or simply run make doc
.
Please refer to the Contributing Guidelines for more details.
Please refer to the community repository.
See the LICENSE file for licensing information.