Object-Granularity Image Segmentation for Precise Language Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language-driven precise instance segmentation methods face challenges in accurately locating instance targets among congested targets of the same category and predicting masks that are not overflowed to adjacent targets, leading to inaccurate target positioning and mask prediction.
Innovation Solution
Introduce a target integrity constraint in cross-modal feature fusion by using neural networks to fuse image features with language encoding at the object granularity, activating instance regions on a per-target basis, and updating the feature extraction network based on the similarity between text features and embedding vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cross-modal feature fusion is used to activate regions correlated with language encoding, then the model can process natural language inputs, but the instance target positioning accuracy deteriorates and masks are overflowed to adjacent targets
Solution Approach 1:
The patent segments the image feature map into multiple candidate regions corresponding to different objects, then processes each region independently through the neural network to generate separate embedding vectors. This segmentation allows the model to distinguish between multiple targets of the same category and assign accurate positions without mask overflow, while still maintaining the ability to process natural language inputs.
2Quantity of substance
If conventional semantic segmentation models are used, then the model can predict labels for targets, but the model cannot distinguish between different targets of the same category
Solution Approach 1:
The patent generates different embedding vectors for different candidate regions, allowing each region to have its own unique representation based on its local visual characteristics and relationship with the language input. This local quality differentiation enables the model to distinguish between different targets of the same category while maintaining comprehensive category coverage.
3Productivity
If language-driven instance segmentation is applied to congested targets, then the model can process multiple targets, but the mask prediction accuracy deteriorates and adjacent targets are affected
Solution Approach 1:
The patent divides the image into multiple candidate regions and processes each region independently through the neural network, generating separate embedding vectors for each. This segmentation approach allows the model to handle multiple congested targets simultaneously while maintaining high mask prediction accuracy for each individual target, preventing mask overflow to adjacent targets.
Data Source
Figure 1A~1C
Figure 1D~2
Figure 3
AI summary
A data processing method is applied to image processing, for example, image segmentation or target detection. The method includes: obtaining a first image feature corresponding to an image and a text feature corresponding to a text; obtaining a plurality of second embedding vectors through a neural network based on a plurality of preset first embedding vectors and the first image feature, where each second embedding vector corresponds to one candidate region of a target object, and each second embedding vector and the first image feature are used to be fused to obtain one corresponding second image feature; and determining, based on a similarity between the text feature and the plurality of second embedding vectors, a weight corresponding to each second embedding vector, where a plurality of weights are used to be fused with t a plurality of second image features, to determine a prediction region corresponding to the target object. In this application, the image feature is changed from being at a pixel granularity to being at a target object granularity, and pixels belonging to a same target are considered as a whole to be fused with language encoding, so that processing precision of a model can be improved.