Open Vocabulary Image Segmentation Using Text-Image Embedding Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional segmentation models lack class awareness and object or instance awareness, resulting in masks that do not accurately represent the class of objects and often generate multiple masks for a single object.
Innovation Solution
A processor-implemented method that uses a pre-trained segmentation model combined with a newly trained decoder to perform panoptic segmentation. The method involves processing an input image to generate image embeddings, using a mask decoder to generate mask embeddings with class and instance awareness, and aggregating these with text embeddings to generate augmented masks with associated classifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional segmentation models are used to generate masks, then the masks can be generated quickly, but the masks lack class awareness and object awareness, resulting in inaccurate segmentation
Solution Approach 1:
The patent combines a pre-trained segmentation model with a newly trained decoder to create a hybrid system that leverages the speed of conventional models while adding class and object awareness through the new decoder component
Solution Approach 2:
The patent uses a pre-trained segmentation model that has already been trained on vast amounts of data to provide a foundation of accurate mask generation, then adds the new decoder trained on specific panoptic segmentation tasks to enhance this pre-established capability
2Loss of information
If conventional segmentation models generate masks without class information, then the processing is simpler, but the masks do not represent the class of objects accurately
Solution Approach 1:
The patent introduces a text encoder as an intermediary that processes textual input and generates text embeddings, which are then aggregated with mask embeddings to provide class information without directly modifying the core segmentation model
Solution Approach 2:
The patent adds a textual dimension to the segmentation process by incorporating text embeddings that represent class information, transforming the problem from purely visual to multi-modal (visual + textual)
3Reliability
If the model generates multiple masks for a single object, then more potential segments are covered, but the masks do not accurately represent individual objects or instances
Solution Approach 1:
The patent employs a decoder that receives feedback from both the image encoder and text encoder, allowing it to refine mask generation by considering both visual features and class information to produce more accurate instance-level segmentation
Solution Approach 2:
The patent uses a dynamic decoding process that adapts based on the aggregated embeddings from multiple sources (image and text), allowing the model to flexibly generate the appropriate number of masks based on the specific input rather than following a fixed pattern
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for improved machine learning. In an example method, an input image is accessed, and the input image is processed using an image encoder to generate an image embedding tensor. The image embedding tensor is processed using a mask decoder machine learning model to generate a set of mask embedding tensors. A textual input is processed using a text encoder to generate a text embedding tensor. A set of augmented masks is generated based on aggregating the text embedding tensor with the set of mask embedding tensors.