Attention Fusion Network for Image Captioning Accuracy
Solution Overview
Problem
Current image caption generation methods primarily focus on visual features, lacking comprehensive integration of semantic features, which can lead to incomplete or inaccurate image descriptions.
Innovation Solution
An information generating method and apparatus that extracts both semantic and visual feature sets from an image and employs an attention fusion network to combine these features at multiple time steps, using a visual-semantic double attention model to generate accurate image captions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If only visual features are used for image caption generation, then the method is simple and fast, but the accuracy and completeness of image descriptions are insufficient
Solution Approach 1:
The patent combines visual features and semantic features into a unified attention fusion network. The visual encoder extracts visual features while the semantic encoder extracts semantic features from the same image, and both feature streams are merged through the attention fusion network to generate comprehensive image captions. This merging approach resolves the contradiction by integrating multiple feature types to improve accuracy while managing complexity through a unified architectural framework.
Solution Approach 2:
The patent creates a composite feature representation by combining visual features and semantic features. The attention fusion network processes both feature types simultaneously, creating a composite understanding of the image that leverages the strengths of both visual and semantic information. This composite approach improves caption accuracy by utilizing multiple complementary feature sources rather than relying on a single feature type.
2Loss of information
If multiple feature types are integrated through attention fusion network, then the completeness of image descriptions improves, but the computational complexity and processing time increase
Solution Approach 1:
The patent performs preliminary feature extraction by parallel processing of visual and semantic features before fusion. The visual encoder and semantic encoder operate simultaneously to extract their respective features, and the attention fusion network then combines these pre-extracted features. This preliminary action approach reduces processing time by avoiding sequential extraction and allows for optimized parallel computation of multiple feature streams.
Solution Approach 2:
The attention fusion network dynamically adjusts the weighting and integration of visual and semantic features based on the specific image content and captioning context. The attention mechanism adaptively determines which feature type is more relevant for each particular caption generation task, optimizing processing efficiency by focusing computational resources on the most informative features for each specific case rather than uniformly processing all features equally.
Data Source
AI summary
An information generating method is performed by a computer device. The method includes: obtaining a target image; extracting a semantic feature set and a visual feature set of the target image; performing attention fusion on semantic features and visual features of the target image at n time steps to obtain caption words of the target image at the n time steps by processing the semantic feature set and the visual feature set of the target image through an attention fusion network in an information generating model; and generating image caption information of the target image based on the caption words of the target image at the n time steps. Through the foregoing method, an advantage of the visual feature in generating visual vocabulary and an advantage of the semantic feature in generating a non-visual feature are combined, thereby improving the image caption's accuracy.