Image Description Generation Using Multi-Mode Feature Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image description generation methods rely solely on global feature vectors and label vectors, resulting in inaccurate image description information due to limited input parameters for decoders.
Innovation Solution
An image description generation method that includes a feature extraction model, a matching model, and a computing model, where the matching model generates a multi-mode feature vector by training on both global and text feature vectors, and the computing model uses this vector along with label vectors to produce more accurate image descriptions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only global feature vectors and label vectors are used as input parameters for decoders, then the system complexity is low, but the image description accuracy is insufficient
Solution Approach 1:
The patent segments the feature extraction process into multiple specialized models: a global feature extraction model, a text feature extraction model, and a matching model. Each model focuses on extracting specific types of features (global spatial features, text features, and matched features respectively), thereby improving description accuracy without overwhelming complexity in a single monolithic model.
Solution Approach 2:
The patent introduces a new dimensional aspect by adding text feature vectors and multi-mode feature vectors to the traditional global feature vectors. This creates a multi-dimensional feature space that enriches the input parameters for the decoder, enabling more accurate image descriptions through enhanced feature representation.
2Measurement precision
If multiple feature vectors are generated through multiple models, then the image description accuracy is improved, but the computational time and resources increase
Solution Approach 1:
The patent performs preliminary feature extraction by generating global feature vectors, text feature vectors, and multi-mode feature vectors before the main decoding process. This preliminary action prepares enriched feature representations in advance, which are then fed into the decoder to improve description accuracy without adding significant computational overhead during the main processing stage.
Solution Approach 2:
The patent merges multiple feature vectors (global feature vectors, text feature vectors, and multi-mode feature vectors) into a unified feature representation that serves as input for the decoder. This merging consolidates information from different sources into a cohesive structure, improving accuracy while streamlining the computational process.
Data Source
AI summary
Embodiments of this application disclose an image description generation method performed at a computing device. The method includes: obtaining a target image; generating a first global feature vector and a first label vector set of the target image; generating a first multi-mode feature vector of the target image through a matching model, the matching model being a model obtained through training according to a training image and reference image description information of the training image; and applying the first multi-mode feature vector, the first global feature vector, and the first label vector set to a computing model, to obtain the target image description information, the computing model being a model obtained through training according to image description information of the training image and the reference image description information.