Neural Network Pixel Offset Vectors for Real-Time Instance Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current instance segmentation methods, such as Mask R-CNN, generate low-resolution masks and operate at a low frame rate, making them impractical for real-time applications like autonomous driving, while proposal-free methods using fully-convolutional networks face accuracy issues due to spatial-invariance and require post-processing steps that hinder end-to-end optimization.
Innovation Solution
A system utilizing a trained neural network that generates pixel offset vectors pointing to a unique object center, forming a cluster distribution with a learnable margin, allowing for real-time instance segmentation with high accuracy and differentiating between similar objects, and employing a dense-prediction network to generate masks at original input resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Mask R-CNN framework is used for instance segmentation, then high segmentation accuracy is achieved, but processing speed is low and resolution is reduced
Solution Approach 1:
The patent segments the instance segmentation task into two independent parts: (1) object detection using a detector that outputs bounding boxes and class labels, and (2) mask generation using a separate U-Net segmentation network. This segmentation allows each component to be optimized independently, achieving both high accuracy and real-time performance.
Solution Approach 2:
The patent performs preliminary object detection and bounding box extraction before feeding the data to the segmentation network. By pre-processing the image to identify object locations and classes beforehand, the segmentation network only needs to generate masks for detected objects, significantly reducing computational load and enabling real-time processing.
2Productivity
If fully-convolutional networks are used for proposal-free instance segmentation, then real-time processing is achieved, but segmentation accuracy deteriorates due to spatial-invariance
Solution Approach 1:
The patent divides the processing pipeline into detection and segmentation stages, allowing the use of convolutional networks for real-time processing while maintaining accuracy through the detection stage that provides spatial information about object locations.
Solution Approach 2:
The patent introduces an intermediary detection stage that provides bounding box annotations and object location information. This intermediary layer bridges the gap between real-time processing capabilities and accurate segmentation, allowing the segmentation network to focus only on regions of interest identified by the detector.
3Ease of operation
If detection-and-segment approach is used, then object detection is simplified, but mask resolution is low and frame rate is reduced
Solution Approach 1:
The patent segments the detection and segmentation tasks into separate modules, allowing the detection module to maintain simplicity while the segmentation module (U-Net) generates high-resolution masks. The modular architecture enables each component to be optimized for its specific function.
Solution Approach 2:
The patent performs preliminary detection to identify object bounding boxes before segmentation. This preliminary action provides a guide for the segmentation network, allowing it to allocate computational resources efficiently and generate high-resolution masks only for detected objects rather than processing the entire image at full resolution.
Data Source
AI summary
A system for generating a mask for object instances in an image is provided. The system includes a first module comprising a trained neural network and configured to input the image to the neural network, wherein the neural network is configured to generate: pixel offset vectors for the pixels of the object instance configured to point towards a unique center of an object instance, the pixel offset vectors thereby forming a cluster with a cluster distribution, and for each object instance an estimate of said cluster distribution defining a margin for determining which pixels belong to the object instance. A method for training a neural network map to be used for generating a mask for object instances in an image is also provided.