fuzzy-search / realtime-bakllava Goto Github PK

llama.cpp with BakLLaVA model describes what does it see

Python 100.00%

cpp demo-application inference llama llamacpp llm bakllavva

realtime-bakllava's Introduction

🍰 Bakllava Llama C++ Tutorial 🦙

Welcome to the delicious world of Bakllava Llama with C++! Follow these steps to get your code running and indulge in AI sweetness! 😋

🚨 Properly tested only with Apple silicon chip

youtube installation guide

similar relevant project: Be My Eyes" web app

🚀 Step 1: Install Llama C++

First things first, let's get the Llama C++ installed.

🔗 Clone the repository from GitHub:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

On Linux & macOS:

🛠 Build with make:

make

🏗 Or, if you prefer cmake:

cmake --build . --config Release

📦 Step 2: Download the Model!

📥 Download from Hugging Face - mys/ggml_bakllava-1 this 2 files:

🌟 ggml-model-q4_k.gguf (or any other quantized model) - only one is required!
🧊 mmproj-model-f16.gguf

✂️ Copy the paths of those 2 files.

🏃‍♂️ Run this in the llama.cpp repository (replace YOUR_PATH with the paths to the files you downloaded):

macOS

./server -m YOUR_PATH/ggml-model-q4_k.gguf --mmproj YOUR_PATH/mmproj-model-f16.gguf -ngl 1

Windows

server.exe -m REPLACE_WITH_YOUR_PATH\ggml-model-q4_k.gguf --mmproj REPLACE_WITH_YOUR_PATH\mmproj-model-f16.gguf -ngl 1

🎉 The llama server is now up and running!

⚠️ NOTE: Keep the server running in the background.
📹 Let's run the script to use the webcam or send it a single picture!

🏃‍♀️ Step 3: Running the Demo

Open a new terminal window and clone the demo app:

git clone https://github.com/Fuzzy-Search/realtime-bakllava.git
cd realtime-bakllava

🛠 (Optional) Create a new Python virtual environment and activate it

python3 -m venv bakllava-venv
source bakllava-venv/bin/activate
pip3 install -r requirements.txt

🎥 Webcam Script

To start streaming from your webcam:

! if you have problem with FFMPEG lib, download the source code and in file src/video_stream.py modify second line of code

python3 src/video_stream.py

🖼 Simple Picture Drop

pip install -r picture_requirements.txt
python src/picture_drop.py --path src/sample_pic.png

📝 Enjoy your adventure with Llama C++! 🚀🦙

Star History

realtime-bakllava's People

Contributors

Stargazers

Watchers

realtime-bakllava's Issues

Could not find a backend to open `<video0>`` with iomode `r?`.

When running:

python3 src/video_stream.py

get error:

Starting video stream... Wait for a few seconds for the stream to the output to start generating.
Traceback (most recent call last):
  File "/workspace/realtime-bakllava/src/video_stream.py", line 18, in <module>
    cap = imageio.get_reader('<video0>')
  File "/usr/local/lib/python3.10/dist-packages/imageio/v2.py", line 290, in get_reader
    image_file = imopen(uri, "r" + mode, **imopen_args)
  File "/usr/local/lib/python3.10/dist-packages/imageio/core/imopen.py", line 281, in imopen
    raise err_type(err_msg)
ValueError: Could not find a backend to open `<video0>`` with iomode `r?`.

ffmpeg does not stream the webcam feed to a app window

When I start ffmpeg, I don't see a window of the webcam display. Although the green light goes on to say the webcam is on.

Then the console starts by see black and responses with content like this:

 The image displays a dark background with a deep black color dominating the scene. In this stark environment, there is a single white dot that stands out prominently, serving as the focal point of the composition. Surrounding the white dot, there are nine smaller dots arranged in various positions throughout the space, collectively creating an intriguing visual contrast against the dark backdrop.JSONDecodeError: Expecting property name enclosed in double quotes
 The image features a dark night sky with no visible objects or people. It is a deep black background that creates a sense of emptiness and vacuum-like void. This blackness encompasses the entire frame of the picture, leaving room for only the subtest to provide any contrast or visual interest

Any suggestions on why ffmpeg is not streaming the webcam?

JSONDecodeError: Expecting property name enclosed in double quotes

I'm seeing JSONDecodeError: Expecting property name enclosed in double quotes at the end of each description.

Example:

In addition to the tree and bench, there is a deck nearby with a chair on it, offering another place to appreciate the view of the lake. The combination of the tree, water, and outdoor furniture creates a serene and inviting setting for rest and relaxation.JSONDecodeError: Expecting property name enclosed in double quotes