Retrieval-Augmented Image Segmentation for Open Vocabulary Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing panoptic segmentation systems face challenges in encoding masked images effectively, leading to poor quality features and impaired performance for open vocabulary scenarios, particularly when encountering objects not included in the training dataset.
Innovation Solution
A retrieval-augmented approach is employed, constructing a feature database for masked regions, using CLIP-text and CLIP-vision embeddings to retrieve similar features and class labels, and combining this with CLIP-score classification to enhance open vocabulary panoptic segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If CLIP vision encoder is used to encode masked image regions, then open vocabulary segmentation is enabled, but feature quality deteriorates due to encoding limitations
Solution Approach 1:
The patent introduces an intermediary retrieval system that mediates between the masked region features and class labels. Instead of relying solely on the CLIP vision encoder's direct encoding of masked regions (which produces poor quality features), the system retrieves features from a pre-built database of full-image features, thus bridging the gap between masked region input and high-quality feature representation for open vocabulary segmentation.
Solution Approach 2:
The system performs preliminary action by pre-building a feature database using full natural images before the actual segmentation task. This database contains high-quality features extracted from complete images, which are then retrieved during inference to compensate for the quality loss from encoding masked regions alone.
2Measurement precision
If retrieval augmentation is applied to improve open vocabulary segmentation, then classification accuracy improves, but system complexity increases due to feature database construction
Solution Approach 1:
The feature database is constructed in advance during a preprocessing stage, separating the complexity of database construction from the inference process. This allows the system to achieve high classification accuracy during runtime by simply retrieving pre-computed features, rather than performing complex computations during actual segmentation tasks.
Solution Approach 2:
The system creates copies of high-quality features from full images and stores them in a database. During segmentation, instead of重新 computing features from masked regions, the system retrieves and uses these pre-computed feature copies, which significantly improves classification accuracy while reducing computational complexity during inference.
3Manufacturing precision
If supervised learning is conducted on predefined classes, then closed vocabulary segmentation performance improves, but adaptability to arbitrary classes deteriorates
Solution Approach 1:
The patent makes the system universal by enabling it to handle both closed vocabulary and open vocabulary segmentation tasks. The retrieval-based classification module can work with predefined classes (maintaining high segmentation performance) while also adapting to arbitrary classes through the feature database that stores diverse image features, thus achieving multi-functionality.
Solution Approach 2:
The retrieval system acts as an intermediary that connects the supervised learning component (trained on predefined classes) with the open vocabulary requirement (arbitrary classes). By retrieving features from the database that were learned from diverse data, the system bridges the gap between supervised training on fixed classes and the need to recognize arbitrary classes.
Data Source
AI summary
A system and a method are disclosed for classifying features from an input image. The method includes generating, by a processing circuit, a segment feature from a first input image, the segment feature corresponding to a group of pixels associated with an object represented in the first input image, and being an out-of-vocabulary segment feature; performing, by the processing circuit, a retrieval of a first feature vector, corresponding to the segment feature, from a database of feature vectors, the first feature vector representing an object-specific segmentation mask; and generating, by the processing circuit, an output segmentation mask based on the first feature vector.