Open-Vocabulary Image Segmentation With Negative Mask Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing open-vocabulary semantic segmentation methods struggle to accurately segment regions of interest using unseen categories, often producing false positive predictions, and existing few-shot methods are limited to closed-set segmentation, hindering real-world applicability.
Innovation Solution
A personalized open-vocabulary semantic segmentation approach is introduced, utilizing a negative mask proposal and incorporating visual embeddings from a pre-trained image encoder to enhance the segmentation process, reducing false positives and improving accuracy by focusing on regions of user interest.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If open-vocabulary semantic segmentation methods are used to segment unseen categories, then the versatility and applicability of the segmentation system is improved, but false positive predictions increase and measurement precision deteriorates
Solution Approach 1:
The patent divides the segmentation task into multiple stages: generating mask proposals from image features, filtering proposals using textual embeddings and prompts, and producing final segmentations. This multi-stage segmentation approach allows the system to handle unseen categories while maintaining precision through progressive filtering
Solution Approach 2:
The patent introduces textual embeddings and textual prompts as intermediary elements between image features and final segmentation masks. These intermediaries enable the system to reason about unseen categories through language descriptions while maintaining control over false positives through the matching process
2Productivity
If few-shot segmentation methods are used to limit training data, then the efficiency of the system is improved, but the system is restricted to closed-set segmentation and adaptability deteriorates
Solution Approach 1:
The patent creates a universal segmentation framework that can handle both seen and unseen categories through the open-vocabulary mechanism. The system uses textual embeddings to represent both training categories and test-time queries, enabling the same model to perform few-shot segmentation for novel categories without retraining
Solution Approach 2:
The patent changes the parameter space from fixed category IDs to continuous textual embeddings. This allows the system to adapt to new categories by simply providing text descriptions rather than requiring retraining, thereby maintaining efficiency while expanding adaptability to open-vocabulary scenarios
3Measurement precision
If traditional semantic segmentation methods are used with fixed categories, then measurement precision is improved, but device complexity increases to manage category mappings and limitations
Solution Approach 1:
The patent extracts the category management burden from the model architecture by using textual embeddings stored in an external vocabulary. Instead of hardcoding category mappings in the network, the system separates the visual feature extraction from the category identification, reducing internal complexity while maintaining precision
Data Source
AI summary
Disclosed are systems and techniques for image processing. For example, a computing device can process, using an encoder, an image to generate a feature map representing the image. The computing device can use the encoder to determine, based on the feature map, mask embeddings, negative mask embeddings, textual embeddings, and textual prompts for semantic segmentation of the image. The computing device can use a semantic segmentation model to determine, based on the feature map, mask proposals and a negative mask for the image and to determine a similarity map between total mask embeddings (including the mask embeddings and the negative mask embeddings) and total textual embeddings (including the textual embeddings and the textual prompts). The computing device can determine, using the semantic segmentation model, final semantic predictions for the image based on the similarity map and total mask proposals (including the mask proposals and the negative mask).