PaLM-Kosmos-Vision is a proof-of-concept web application that serves as a basic version of ChatGPT Vision multi-modal chat. It's designed to demonstrate that integrating MultiModal models with traditional LLMs is possible, laying the groundwork for more sophisticated applications. PaLM-Kosmos-Vision is an innovative web application that combines the capabilities of Large Language Models (LLMs) to describe images and engage users in detailed conversations about those images. This project showcases the integration of PaLM API for nuanced language understanding and the KOSMOS-2 model for generating image captions.

• Image upload functionality with real-time display.
• Automatic image captioning using the KOSMOS-2 model.
• Detailed chat interactions about images powered by PaLM API.
• Session-based chat history with a clear chat option.
• Demo is limited to 20 messages per user to optimize resource usage.

• Streamlit as the web app framework.
• PaLM and KOSMOS-2 models for AI functionalities.
• Replicate for model deployment and serving.
• LlamaIndex for efficient LLM orchestration.
• Python libraries such as requests, Pillow, transformers.

Clone the repository and install the necessary Python packages. Ensure to set up API keys for Google Cloud and Replicate as environment variables before running the application.

Start the app with Streamlit and upload an image to begin. The interface allows for interactive conversations about the image, with the app providing contextually relevant information.

Try out the live demo here: PaLM-Kosmos-Vision Demo

If you're interested in contributing, please review the contribution guidelines. Your ideas and pull requests are welcome!

This project is licensed under the MIT License.

Developed by Harshad Suryawanshi. Connect with me on LinkedIn.