Cross-Media Retrieval Using Deep Semantic Space Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cross-media retrieval methods fail to effectively bridge the 'perception gap' between visual features and high-level user concepts, leading to low accuracy in Image Retrieval in Text (Img2Text) and Text Retrieval in Image (Text2Img) tasks due to the lack of integration of visual and linguistic information.
Innovation Solution
A cross-media retrieval method based on deep semantic space is proposed, utilizing a feature generation process and semantic space learning process that simulates human perception by generating CNN visual feature vectors and LSTM language description vectors, and employing Multi-Sensory Fusion Deep Neural Networks (MSF-DNN) and TextNet to map images and texts into an isomorphic semantic space for improved retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If heterogeneous features are mapped to homogeneous space to span the semantic gap, then cross-media retrieval becomes feasible, but the perception gap between visual features and high-level user concepts is ignored resulting in low accuracy
Solution Approach 1:
The patent segments the feature extraction process into distinct components: visual feature extraction from images, linguistic feature extraction from texts, and topic information extraction. By separating these extraction processes and then integrating them in a unified semantic space, the system maintains the unique characteristics of each modality while enabling cross-media retrieval, thus resolving the contradiction between versatility and accuracy.
Solution Approach 2:
The patent creates a composite feature representation by combining visual features, linguistic features, and topic information into a unified semantic space. This composite approach integrates multiple types of information (images, texts, topics) to form a richer representation that bridges the perception gap while maintaining cross-media retrieval capability.
2Device complexity
If visual and linguistic information are processed separately in existing methods, then processing is simpler, but the association between underlying visual features and high-level user concepts cannot be established
Solution Approach 1:
The patent merges visual feature processing and linguistic feature processing into a unified semantic space where images, texts, and topics can be associated. By combining these processing streams in a shared feature space rather than keeping them separate, the system establishes meaningful associations between visual features and high-level concepts while maintaining manageable processing complexity through modular architecture.
3Productivity
If semantic information representation is simplified in the isomorphic space, then computation is faster, but semantic information representation of images and texts is missing resulting in low retrieval performance
Solution Approach 1:
The patent extracts and preserves key semantic information from both visual and linguistic features when mapping to the isomorphic space. By selectively extracting the most important semantic components and maintaining their relationships in the unified space, the system achieves efficient computation without losing critical semantic representation accuracy needed for high retrieval performance.
Data Source
AI summary
The present application discloses a cross-media retrieval method based on deep semantic space, which includes a feature generation stage and a semantic space learning stage. In the feature generation stage, a CNN visual feature vector and an LSTM language description vector of an image are generated by simulating a perception process of a person for the image; and topic information about a text is explored by using an LDA topic model, thus extracting an LDA text topic vector. In the semantic space learning phase, a training set image is trained to obtain a four-layer Multi-Sensory Fusion Deep Neural Network, and a training set text is trained to obtain a three-layer text semantic network, respectively. Finally, a test image and a text are respectively mapped to an isomorphic semantic space by using two networks, so as to realize cross-media retrieval. The disclosed method can significantly improve the performance of cross-media retrieval.