Using YOLOv8(seg) with SHAP about ultralytics HOT 5 OPEN

@Flippchen hi there,

Thank you for your kind words about the Ultralytics library! 😊

Integrating SHAP with YOLOv8 for segmentation tasks is indeed an intriguing idea for gaining insights into model behavior. While SHAP primarily supports classification models out-of-the-box, it can be adapted for object detection and segmentation models with some custom adjustments.

To get started, you'll need to create a custom wrapper for your YOLOv8 model that can interface with SHAP. Here's a basic outline to help you set this up:

1. Load your YOLOv8 model:

   from ultralytics import YOLO
   
   # Load a pretrained YOLOv8 segmentation model
   model = YOLO('yolov8n-seg.pt')

2. Define a prediction function:
   This function should take an image and return the model's predictions in a format that SHAP can work with.

   import numpy as np
   
   def yolo_predict(images):
       results = model(images)
       # Extract the segmentation masks or other relevant outputs
       masks = [result.masks for result in results]
       return np.array(masks)

3. Integrate with SHAP:
   Use SHAP's Image masker and Explainer to create explanations for your model's predictions.

   import shap
   
   # Create a masker for images
   masker = shap.maskers.Image("inpaint_telea", (640, 640, 3))
   
   # Create an explainer using the custom prediction function
   explainer = shap.Explainer(yolo_predict, masker)
   
   # Select an image to explain
   image = np.array([shap.datasets.imagenet50()[0]])  # Replace with your image
   
   # Generate SHAP values
   shap_values = explainer(image)
   
   # Visualize the explanation
   shap.image_plot(shap_values, image)

This is a simplified example to get you started. You might need to adjust the prediction function to better suit your specific needs, especially if you want to focus on particular aspects of the segmentation output.

If you encounter any issues or need further assistance, please ensure you are using the latest versions of torch and ultralytics packages. You can update them using:

pip install --upgrade torch ultralytics

For more detailed guidance, you can refer to the SHAP documentation and examples you mentioned. They provide a good foundation for adapting SHAP to different model architectures.

Feel free to reach out if you have any more questions. Happy coding! 🚀

from ultralytics.

@glenn-jocher Thank you for your quick reply and help.

Unfortunately the example does not work directly and I cannot get it to work in general.

The problem seems to be that shap assumes that the first dimension of the image provides a stack dimension. Because when I use this I get the error
raise DimensionError("The length of the image to be masked must match the shape given in the " + \ shap.utils._exceptions.DimensionError: The length of the image to be masked must match the shape given in the ImageMasker constructor: 640 * 3 != 640 * 640 * 3 where it seems to lose the first 640 of the image.

When I add a batch dimension to the image with np.expand(image, dim=0) I get an error in the ultralytics library in augment.py. I think because the image has too much dimension. Error: line 774, in __call__'. img = cv2.copyMakeBorder( cv2.error: OpenCV(4.9.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\copy.cpp:1026: Error: (-215:Assertion failed) top >= 0 && bottom >= 0 && left >= 0 && right >= 0 && _src.dims() <= 2 in function 'cv::copyMakeBorder'.

My sample code:

model = YOLO("path_to_file.pt")

def yolo_predict(images):
    results = model(images)
    # Extract the segmentation masks or other relevant output
    masks = [result.masks for result in results] return np.array(masks)
    return np.array(masks)


# Create a masker for the images
masker = shap.maskers.Image("inpaint_telea", (640, 640, 3))

# Create an explainer using the custom prediction function
explainer = shap.Explainer(yolo_predict, masker)

# Select an image to explain
image_path = r"path_to_image.png"
img = Image.open(image_path)
img = img.resize((640, 640))
image = np.array(img)  
image_with_batch = np.expand_dims(image, axis=0)
print(image_with_batch.shape)

# Generate SHAP values
shap_values = explainer(
    image_with_batch, max_evals=20, outputs=shap.Explanation.argsort.flip[:1])
)
# Visualise the explanation
shap.image_plot(shap_values, image)

Can you help here? Maybe the model must be loaded directly via pytorch?

from ultralytics.

Hi @Flippchen,

Thank you for providing the detailed code and error messages. Let's address the issues you're encountering with SHAP and YOLOv8.

The error you're seeing is due to the mismatch in expected dimensions by SHAP and the YOLOv8 model. SHAP expects the input image to have a batch dimension, but it seems there's a conflict when passing this to the YOLOv8 model.

Here's a revised version of your code to ensure compatibility:

1. Ensure the image has the correct dimensions:

   • SHAP expects the input to have a batch dimension.
   • YOLOv8 expects the input to be in the correct format for its processing.

2. Adjust the prediction function to handle the batch dimension correctly:

   • Ensure the input image is processed correctly by the YOLOv8 model.

Here's the updated code:

import numpy as np
from PIL import Image
import shap
from ultralytics import YOLO

# Load the YOLOv8 segmentation model
model = YOLO("path_to_file.pt")

def yolo_predict(images):
    # Ensure the input is in the correct format for YOLOv8
    images = [np.array(image) for image in images]
    results = model(images)
    # Extract the segmentation masks or other relevant outputs
    masks = [result.masks for result in results]
    return np.array(masks)

# Create a masker for the images
masker = shap.maskers.Image("inpaint_telea", (640, 640, 3))

# Create an explainer using the custom prediction function
explainer = shap.Explainer(yolo_predict, masker)

# Select an image to explain
image_path = r"path_to_image.png"
img = Image.open(image_path)
img = img.resize((640, 640))
image = np.array(img)
image_with_batch = np.expand_dims(image, axis=0)
print(image_with_batch.shape)

# Generate SHAP values
shap_values = explainer(image_with_batch, max_evals=20, outputs=shap.Explanation.argsort.flip[:1])

# Visualize the explanation
shap.image_plot(shap_values, image_with_batch)

Key Changes:

1. Prediction Function: The yolo_predict function now ensures that each image in the batch is converted to a numpy array before passing it to the YOLOv8 model.
2. Image Preparation: The image is resized and expanded to include the batch dimension before being passed to the SHAP explainer.

Additional Steps:

• Ensure Latest Versions: Make sure you are using the latest versions of torch and ultralytics. You can update them using:

  pip install --upgrade torch ultralytics

If you continue to experience issues, please provide any additional error messages or details. This will help us further diagnose and resolve the problem.

Feel free to reach out if you have any more questions. Happy coding! 🚀

from ultralytics.

Hi @glenn-jocher,

Unfortunately the example provided does not work. Do you have any other suggestions? Can you provide an example using onnx and the raw output of the model for example?

from ultralytics.

Hi @Flippchen,

Thank you for your patience. Let's explore another approach using ONNX and the raw output of the model.

First, ensure you have the latest versions of torch and ultralytics installed:

pip install --upgrade torch ultralytics

Next, let's export your YOLOv8 model to ONNX format and use it for predictions. Here's how you can do it:

1. Export the YOLOv8 model to ONNX:

   from ultralytics import YOLO
   
   # Load the YOLOv8 segmentation model
   model = YOLO("path_to_file.pt")
   
   # Export the model to ONNX format
   model.export(format="onnx")

2. Load the ONNX model and make predictions:

   import onnxruntime as ort
   import numpy as np
   from PIL import Image
   
   # Load the ONNX model
   ort_session = ort.InferenceSession("yolov8n.onnx")
   
   def to_numpy(tensor):
       return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()
   
   def yolo_predict(images):
       # Preprocess the images
       images = [np.array(image) for image in images]
       images = np.stack(images, axis=0)
   
       # Run the ONNX model
       ort_inputs = {ort_session.get_inputs()[0].name: images}
       ort_outs = ort_session.run(None, ort_inputs)
   
       # Extract the segmentation masks or other relevant outputs
       masks = ort_outs[0]  # Adjust based on your model's output structure
       return masks
   
   # Create a masker for the images
   masker = shap.maskers.Image("inpaint_telea", (640, 640, 3))
   
   # Create an explainer using the custom prediction function
   explainer = shap.Explainer(yolo_predict, masker)
   
   # Select an image to explain
   image_path = r"path_to_image.png"
   img = Image.open(image_path)
   img = img.resize((640, 640))
   image = np.array(img)
   image_with_batch = np.expand_dims(image, axis=0)
   print(image_with_batch.shape)
   
   # Generate SHAP values
   shap_values = explainer(image_with_batch, max_evals=20, outputs=shap.Explanation.argsort.flip[:1])
   
   # Visualize the explanation
   shap.image_plot(shap_values, image_with_batch)

This approach uses ONNX for inference, which might provide better compatibility with SHAP. If you encounter any issues, please provide a minimum reproducible example so we can investigate further. You can find more details on creating a reproducible example here.

Feel free to reach out if you have any more questions. Happy coding! 🚀

from ultralytics.