Hardware-accelerated, deep learned semantic image segmentation

Isaac ROS Image Segmentation contains a ROS 2 package to produce semantic image segmentation. isaac_ros_unet provides a method for classification of an input image at the pixel level. Each pixel of the input image is predicted to belong to a set of defined classes. Classification is performed with GPU acceleration running DNN inference on a U-NET architecture model. The output prediction can be used by perception functions to understand where each class is spatially in a 2D image or fuse with a corresponding depth location in a 3D scene.

A trained model based on the U-NET architecture is required to produce a segmentation mask. Input images may need to be cropped and resized to maintain the aspect ratio and match the input resolution of the U-NET DNN; image resolution may be reduced to improve DNN inference performance, which typically scales directly with the number of pixels in the image.

Image segmentation provides more information and uses more compute than object detection to produce classifications per pixel, whereas object detection classifies a simpler bounding box rectangle in image coordinates. Object detection is used to know if, and where spatially in a 2D image, the object exists. On the other hand, image segmentation is used to know which pixels belong to the class. One application is using the segmentation result, and fusing it with the corresponding depth information in order to know an object location in a 3D scene.

This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.

Please visit the Isaac ROS Documentation to learn how to use this repository.

• isaac_ros_unet
  • Quickstart
  • Try More Examples
  • Troubleshooting
  • API

Update 2023-10-18: Updated for Isaac ROS 2.0.0.