Welcome to this GitHub repository. Here, we provide example scripts to deploy different Huggingface models on Databricks Model Serving. These examples can also guide you in deploying other models following similar steps.

We suggest beginning with the following script. The first notebook uses the "mlflow transformer" flavor to demonstrate the ease and simplicity of deploying models. The second notebook uses "mlflow pyfunc" to illustrate how you can pass additional parameters or can add pre-processing/post-processing with the deployed models.

• GPT2 deployment using mlflow transformer flavor
• GPT2 deployment with mlflow pyfunc

Optimized LLM Serving enables you to take state of the art OSS LLMs and deploy them on Databricks Model Serving with automatic optimizations for improved latency and throughput on GPUs. Currently, we support optimizing the Mosaic MPT model and will continue introducing more models with optimization support.

• Optimized LLM deployment with mpt-instruct

You can quantize models to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the 16-bit binary floating point (bfloat16). With quantization, you can deploy 13b model on single A10 and a 7b model on T4 GPU.

Note: Quantizing the model can degrade model performance and may not necessarily make it faster.

• mpt-7b-instruct deployment with 8-bit quantization

Please refer to this repository for scripts that detail how to fine-tune LLMs on Databricks: https://github.com/databricks/databricks-ml-examples.

Before you start, please ensure you meet the following requirements:

• Ensure that you have Nvidia A10/A100 GPUs to run the script.

• Ensure that you have MLflow 2.3+ (MLR 13.1 beta) installed.

• Deployment requires GPU model serving. For more information on GPU model serving, contact the Databricks team or sign up here.

• Here are some general guidelines for determining GPU requirements when serving a model.

Clone this repository and navigate to the desired script file. Follow the instructions within the script to deploy the model, ensuring you meet the requirements listed above.

Feel free to contribute to this project by forking this repo and creating pull requests. If you encounter any issues or have any questions, create an issue on this repo, and we'll try our best to respond in a timely manner.

This project is licensed under the terms of the MIT license. For the usage license of the individual models, please check the respective links provided above.