Bidirectional Attention Image-Text Cross-Modal Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image-text cross-modal retrieval methods face challenges in accurately capturing shared semantics between images and text due to the heterogeneity of data modalities, leading to limited feasibility in measuring similarity between higher-order features and affecting retrieval precision.
Innovation Solution
A bidirectional attention-based method is employed, using a deep neural network to extract initial features and reconstruct them with a bidirectional attention module, capturing joint features and weighting them to explore interplay information, thereby enhancing semantic capture and similarity measurement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a simple deep neural network is used to extract features from images and texts, then the device complexity is reduced, but the measurement precision of similarity between higher-order features deteriorates due to data heterogeneity
Solution Approach 1:
The patent introduces an attention mechanism as an intermediary component between the deep neural network feature extraction and the similarity calculation. This attention mechanism processes the heterogeneous image and text features separately before combining them, enabling precise similarity measurement without increasing the overall network structure complexity. The attention mechanism acts as a mediator that handles the data heterogeneity issue while maintaining computational efficiency.
2Device complexity
If only text features are reconstructed using unidirectional attention from image features, then the device complexity is reduced, but the reliability of feature correspondence deteriorates because image and text interplay is not fully captured
Solution Approach 1:
The patent applies bidirectional attention mechanism where both image features and text features are reconstructed from each other, rather than only reconstructing text from image. This inversion of the unidirectional approach ensures that the interplay between image and text is fully captured in both directions, significantly improving the reliability of feature correspondence while maintaining reasonable device complexity.
3Reliability
If bidirectional attention is used to reconstruct both image and text features, then the reliability of feature correspondence is improved, but the device complexity increases due to the dual-directional attention modules
Solution Approach 1:
The patent implements bidirectional attention by applying the same attention mechanism structure to both image-to-text and text-to-image reconstruction, using consistent parameter initialization and update rules. This parameter-based approach ensures that the reliability of feature correspondence is improved through bidirectional processing while controlling device complexity through parameter sharing and consistent architectural design.
Data Source
AI summary
The present disclosure provides a bidirectional attention-based image-text cross-modal retrieval method, applicable for cross-modal retrieval between natural image and electronic text. The present disclosure extracts initial image and text features by using a neural network, and builds a bidirectional attention module to reconstruct the initial image and text features extracted by the neural network, the reconstructed features containing richer semantic information. By using the bidirectional attention module, the present disclosure improves the conventional feature extraction process, obtaining higher-order features with richer image and text semantics, thereby realizing image-text cross-modal retrieval.