Precise Language-Driven Instance Segmentation via Target-Level Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language-driven precise instance segmentation methods face challenges in accurately locating instance targets among congested targets of the same category and predicting masks that are not overflowed to adjacent targets, leading to inaccurate target positioning and mask prediction.
Innovation Solution
Introduce a target integrity constraint in cross-modal feature fusion by using neural networks to fuse image features with language encoding at the object granularity, activating instance regions on a per-target basis, and determining weights based on similarity between text features and embedding vectors to improve prediction precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cross-modal feature fusion is used to activate regions correlated with language encoding, then language-driven instance segmentation can be performed, but target positioning accuracy deteriorates and mask prediction becomes inaccurate
Solution Approach 1:
The patent segments the image feature map into multiple regions based on object granularity, creating separate embedding vectors for different objects. This allows the model to distinguish between multiple targets of the same category by processing them as independent units, thereby improving target positioning accuracy while maintaining language-driven segmentation capability
Solution Approach 2:
The patent applies local quality by generating different embedding vectors for different objects within the same image, even if they belong to the same category. Each object receives a customized embedding vector that captures its specific characteristics, enabling precise localization and accurate mask prediction for each individual target
2Device complexity
If conventional semantic segmentation models are used, then processing is simplified, but the ability to distinguish between different targets of the same category is lost
Solution Approach 1:
The patent divides the feature processing into object-level segments, where each object is represented by a separate embedding vector. This segmentation approach maintains relative simplicity in processing while enabling the model to distinguish between different instances of the same category, resolving the contradiction between simplicity and precision
Solution Approach 2:
The patent introduces embedding vectors as intermediary representations that bridge the gap between simple semantic segmentation and complex instance-level processing. These embedding vectors serve as mediators that capture object-specific information without requiring complete redesign of the segmentation architecture, thus maintaining simplicity while achieving instance distinction
Data Source
AI summary
A data processing method is applied to image processing. The method includes: obtaining a first image feature corresponding to an image and a text feature corresponding to a text; obtaining a plurality of second embedding vectors through a neural network based on a plurality of preset first embedding vectors and the first image feature, where each second embedding vector corresponds to one candidate region of a target object, and each second embedding vector and the first image feature are used to be fused to obtain one corresponding second image feature; and determining, based on a similarity between the text feature and the plurality of second embedding vectors, a weight corresponding to each second embedding vector, where a plurality of weights are used to be fused with a plurality of second image features, to determine a prediction region corresponding to the target object.