Tracking with 1 model and N multi-stream about ultralytics HOT 9 OPEN

Hello! It looks like you're trying to track multiple video streams independently using a single model instance. Currently, the model.track() function treats all inputs as part of a single video sequence, which is why you're seeing the behavior you described.

To achieve independent tracking for each video feed, you would need to manage separate tracker instances for each stream. Unfortunately, this functionality isn't directly supported in the current implementation of YOLOv8's tracking API, where a single call to model.track() is designed to handle sequential frames from the same video.

A potential workaround is to create separate model instances for each stream and run them in parallel, either using threading or multiprocessing to handle each video feed independently. Here's a simplified example using threading:

import threading
from ultralytics import YOLO

def track_stream(capture, model_path):
    model = YOLO(model_path)
    while True:
        frame = capture.read()[1]
        result = model.track([frame], persist=True)
        # Process result

# Setup video captures
captures = [cv2.VideoCapture(path) for path in video_paths]

# Start tracking threads
threads = []
for capture in captures:
    t = threading.Thread(target=track_stream, args=(capture, 'yolov8n.pt'))
    t.start()
    threads.append(t)

for t in threads:
    t.join()

This example assumes you have a separate cv2.VideoCapture for each video feed. Each thread handles tracking for one video stream using its own model instance.

I hope this helps! Let me know if you have further questions.

from ultralytics.

@abelBEDOYA you're right, using multiple model instances does increase VRAM usage significantly. Unfortunately, YOLOv8's current implementation doesn't support independent tracking for multiple streams with a single model instance.

A potential solution is to use a single model for inference and manage separate trackers manually. Here's a simplified example:

from ultralytics import YOLO
import cv2

# Initialize model
model = YOLO('yolov8n.pt')

# Initialize video captures
captures = [cv2.VideoCapture(path) for path in video_paths]

# Initialize trackers
trackers = [YOLO('yolov8n.pt') for _ in captures]

while True:
    frames = [capture.read()[1] for capture in captures]
    results = model(frames)  # Single model inference

    for i, result in enumerate(results):
        trackers[i].track([result], persist=True)  # Independent tracking
        # Process tracker results

    # Display or further process results

This way, you only load the model once, but maintain separate trackers for each video feed. Hope this helps! 😊

from ultralytics.

Thanks for the response! It's a good option. I have tried it and, as you say, the detections and tracking are independent. However, if you pay attention to VRAM usage, it increases the same amount as it would do if you load several models. To be precise, the VRAM used memory increases approximately n_threads * single_model_size (my GPU memory only allows loading 10 models yolov8m doing this) .

Is there a way to do so without loading several models in the VRAM?

At the end of the day they are necessary several trackers but just one model.

from ultralytics.

Looks good! But I've tried it and the method model.track() doesn't support: [result], result, result[0],... It is meant to receive a string path, a numpy array, camera direction, etc. The error:

Besides, as far as I know, BOT-Sort tracker takes into acount some image features of the detection and I suppose that if you only parse the detection result (bboxes) to do tracking with model.track(), it will rely on a "distances" or "IoU" criterion for matching detections from one frame to another, rather than using high-level features.

from ultralytics.

@abelBEDOYA thanks for pointing that out! You're correct that model.track() expects a path or image input. To handle multiple streams with a single model and independent trackers, you can manually manage the trackers. Here's a revised approach:

1. Perform inference with the model.
2. Update each tracker with the detection results.

Here's an example:

from ultralytics import YOLO
import cv2

# Initialize model
model = YOLO('yolov8n.pt')

# Initialize video captures
captures = [cv2.VideoCapture(path) for path in video_paths]

# Initialize trackers
trackers = [YOLO('yolov8n.pt') for _ in captures]

while True:
    frames = [capture.read()[1] for capture in captures]
    results = model(frames)  # Single model inference

    for i, frame in enumerate(frames):
        trackers[i].track(frame, persist=True)  # Independent tracking
        # Process tracker results

    # Display or further process results

This way, you use a single model for inference and maintain independent trackers for each video feed. Let me know if this helps! 😊

from ultralytics.

@abelBEDOYA thanks for pointing that out! You're correct that model.track() expects a path or image input. To handle multiple streams with a single model and independent trackers, you can manually manage the trackers. Here's a revised approach:

1. Perform inference with the model.

2. Update each tracker with the detection results.

Here's an example:

from ultralytics import YOLO
import cv2

# Initialize model
model = YOLO('yolov8n.pt')

# Initialize video captures
captures = [cv2.VideoCapture(path) for path in video_paths]

# Initialize trackers
trackers = [YOLO('yolov8n.pt') for _ in captures]

while True:
    frames = [capture.read()[1] for capture in captures]
    results = model(frames)  # Single model inference

    for i, frame in enumerate(frames):
        trackers[i].track(frame, persist=True)  # Independent tracking
        # Process tracker results

    # Display or further process results

This way, you use a single model for inference and maintain independent trackers for each video feed. Let me know if this helps! 😊

As I said, doing so, they are loaded several models in GPU memory, as many as capture objects or video feeds. It would be great to have just one model in GPU. How can it be done? I've seen some threading solutions but it is loaded a model in GPU for each thread and I run into the same memory problem...

from ultralytics.

Thank you for your feedback, @abelBEDOYA! You're right; the approach I suggested does indeed load multiple models into GPU memory, which isn't ideal for your use case.

To achieve independent tracking for each video feed while using only one model in GPU memory, you can manually manage the tracking logic. Here's a more refined approach:

1. Perform inference with the single model.
2. Use separate tracker instances for each video feed, but only load the model once.

Here's an example:

from ultralytics import YOLO
import cv2
from collections import defaultdict

# Initialize model
model = YOLO('yolov8n.pt')

# Initialize video captures
captures = [cv2.VideoCapture(path) for path in video_paths]

# Initialize trackers
trackers = [defaultdict(lambda: None) for _ in captures]

while True:
    frames = [capture.read()[1] for capture in captures]
    results = model(frames)  # Single model inference

    for i, result in enumerate(results):
        # Extract detections for the current frame
        detections = result.boxes.xywh.cpu().numpy()
        confidences = result.boxes.conf.cpu().numpy()
        class_ids = result.boxes.cls.cpu().numpy()

        # Update the tracker for each video feed
        if trackers[i]['tracker'] is None:
            trackers[i]['tracker'] = YOLO('yolov8n.pt')  # Initialize tracker
        trackers[i]['tracker'].update(detections, confidences, class_ids)

        # Process tracker results
        # ...

    # Display or further process results

This way, you only load the model once into GPU memory and manage independent trackers for each video feed. Let me know if this helps! 😊

from ultralytics.

A new error shows up... model object has no update() method...

The only way I can imagine is using other tracker, rather than using YOLO trackers. Supervision from Roboflow for example...

from ultralytics.

Thank you for pointing that out! You're correct; the model object in YOLOv8 does not have an update() method. My apologies for the confusion.

To achieve independent tracking for each video feed while using a single model instance, you can manually manage the tracking logic without relying on the update() method. Here's a revised approach:

1. Perform inference with the single model.
2. Use separate tracking logic for each video feed.

Here's an example:

from ultralytics import YOLO
import cv2
from collections import defaultdict

# Initialize model
model = YOLO('yolov8n.pt')

# Initialize video captures
captures = [cv2.VideoCapture(path) for path in video_paths]

# Initialize trackers
trackers = [defaultdict(lambda: None) for _ in captures]

while True:
    frames = [capture.read()[1] for capture in captures]
    results = model(frames)  # Single model inference

    for i, result in enumerate(results):
        # Extract detections for the current frame
        detections = result.boxes.xywh.cpu().numpy()
        confidences = result.boxes.conf.cpu().numpy()
        class_ids = result.boxes.cls.cpu().numpy()

        # Update the tracker for each video feed
        if trackers[i]['tracker'] is None:
            trackers[i]['tracker'] = YOLO('yolov8n.pt')  # Initialize tracker
        # Implement your custom tracking logic here

        # Process tracker results
        # ...

    # Display or further process results

If you prefer using a different tracker like Supervision from Roboflow, that could also be a viable solution. Feel free to integrate any tracker that suits your needs

from ultralytics.