Build, test and share your pipelines - No code required!

💧 Instill VDP (Versatile Data Pipeline) is a source-available tool, designed to streamline your data processing pipelines from inception to completion. If your goal is to develop versatile AI features using Large Language Models (LLM), Generative AI, Vision, or Audio models, Instill VDP empowers you to:

• Effortlessly connect to your unstructured data
• Build pipelines to enable diverse AI functionalities in your applications
• Visually test pipelines with a single click, viewing output at each stage
• Easily share your pipelines to showcase your work

☁️ Instill Cloud offers a fully managed public cloud service, providing you with access to all the fantastic features of Instill VDP without the burden of infrastructure management.

• 🚀 Accelerate AI applications by building end-to-end AI-powered pipelines for unstructured data up to 10 times faster
• 🔌 Utilize pre-built connectors to access data from various sources, powerful AI models, and third-party tools
• 🌟 Benefit from a no-code drag-and-drop pipeline builder, enabling quick and customizable application development
• 🪢 Choose between real-time inference (SYNC) and on-demand workload (ASYNC) processing modes
• 🧁 Enjoy a scalable API-first microservice design, offering an excellent developer experience
• ⚡️ Leverage high-performing backends implemented in Go
• 📊 Gain visibility into pipeline performance through a detailed dashboard
• 🤠 Access no-/low-code interfaces, making VDP suitable for every AI and data practitioner

• macOS or Linux - VDP works on macOS or Linux, but does not support Windows yet.

• Docker and Docker Compose - VDP uses Docker Compose (specifically, Compose V2 and Compose specification) to run all services locally. Please install the latest stable Docker and Docker Compose before using VDP.

Note Code in the main branch tracks under-development progress towards the next release and may not work as expected. If you are looking for a stable alpha version, please use latest release.

Execute the following commands to start pre-built images with all the dependencies:

The stable release version

$ git clone -b v0.17.0-alpha https://github.com/instill-ai/vdp.git && cd vdp

# Launch all services
$ make all

The latest version for development

$ git clone https://github.com/instill-ai/vdp.git && cd vdp

# Launch all services
$ make latest PROFILE=all

🚀 That's it! Once all the services are up with health status, the UI is ready to go at http://localhost:3000!

To shut down all running services:

$ make down

Explore our documentation to discover all available deployment options.

Explore our open-source unstructured data infrastructure stack, comprising a collection of source-available projects designed to streamline every aspect of building versatile AI features with unstructured data. Dive into the potential in our documentation.

Open Source Unstructured Data Infrastructure Stack

To dive into Instill Core and Instill Cloud, we provide:

• ✨ Console for non-developers, empowering them to dive into AI applications and process unstructured data without any coding.
• 🧰 CLI and SDKs for developers to seamlessly integrate with their existing data stack in minutes.
  • Instill CLI
  • Python SDK
  • TypeScript SDK
• 🙌 Join our Discord server to exchange ideas about unstructured data processing, AI, MLOps, and get support from the Instill AI team. We're here to support you every step of the way!

• 📔 Documentation - Check out the documentation website.
• 📘 API Reference - Access http://localhost:3001 after make all, or simply run make doc to access the OpenAPI documentation. The gRPC protocols in protobufs provide the single source of truth for the VDP APIs. The genuine protobuf documentation can be found in our Buf Scheme Registry (BSR).

We strongly believe in the power of community collaboration and deeply value your contributions. Head over to our Community repository, the central hub for discussing our open-source projects, raising issues, and sharing your brilliant ideas.

Please refer to the Contributing Guidelines for more details.

Big thanks to these amazing contributors!

See the LICENSE file for licensing information.