Server/Cloud-ready FastChat Docker image based either on Nvdidia CUDA Ubuntu image including:

• Python 3
• Rust
• FastChat
• Vicuna-7B

This image is intented for people who want to run FastChat using on their own server or cloud instance.

Docker Hub prebuilt images are automatically updated on a daily basis.

• Caveats
• Supported tags and respective Dockerfile links
• Docker Environment Variables
  • PORT
• Roadmap
• Convenience Server Scripts (no Docker)
• Contribute
  • Prerequisites
  • Build
  • Run

• ⚠️ The web application is NOT secured. It's up to you to secure your public HTTP port connection.
• You need a GPU server or instance. The default entrypoint uses a single GPU run but you can override it using docker run --entrypoint ... or a Docker Compose file via entrypoint: [...].
• This image is BIG since it includes a 9GB data model.

• latest-cuda12

REQUIRED

This is the server exposed port to emit and listen socket events.

• Add a CPU-only version

This repository provides a few convenience scripts to install and run FastChat without using Docker, for better performances:

• debian.sh: Copy this file in your /home/[username] directory and run it from this directory via sh ./debian.sh.

X is FastChat version:

• latest: clone the main branch from FastChat repository

Z is the variant tag:

• cuda12: NVidia GPUs

• Docker (Desktop or Engine)
• Make (installation depends on your OS)

make build-X-Z

Example: make build-latest-cuda12.

make run-X
# or:
make run-X-Z

Example: make run-latest-cuda12.