Multi-modal Data Retrieval via Shared Weight Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-modal data retrieval methods are inefficient in handling cross-modal retrieval tasks, as they require splicing data of different modalities and extracting features, which is time-consuming and may not meet real-world speed requirements.
Innovation Solution
A method and apparatus that input target retrieval data into a first feature extraction network to acquire a data feature, and then into a second feature extraction network sharing weights across modalities to obtain a target retrieval feature, enabling efficient retrieval across single-modal and cross-modal scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data of different modalities are spliced and features are extracted using related art methods, then matching accuracy can be improved, but retrieval efficiency deteriorates and speed requirements cannot be met
Solution Approach 1:
The patent divides the feature extraction process into two separate networks: a first feature extraction network for extracting features from spliced multi-modal data, and a second feature extraction network for extracting retrieval features from the first network's output. This segmentation allows each network to specialize in specific extraction tasks, improving both accuracy and efficiency by avoiding the computational overhead of traditional end-to-end splicing methods.
Solution Approach 2:
The patent performs preliminary feature extraction by first processing multi-modal data through the first feature extraction network to obtain intermediate features, which are then fed into the second feature extraction network. This preliminary action separates the complex splicing operation from the final retrieval feature extraction, enabling more efficient processing that meets real-time speed requirements while maintaining matching accuracy.
2Measurement precision
If separate feature extraction networks are used for each modality, then retrieval accuracy for single-modal and cross-modal tasks is improved, but the number of parameters and model complexity increase
Solution Approach 1:
The patent merges the feature extraction capabilities across different modalities by designing a unified two-network architecture that processes image, text, and audio data through shared extraction pathways. The first feature extraction network handles multi-modal input uniformly, and the second network extracts retrieval features in a modality-agnostic manner, reducing overall model complexity while maintaining specialized extraction capabilities for each modality.
Solution Approach 2:
The patent creates universal feature extraction networks that can handle multiple modalities (image, text, audio) and multiple retrieval types (single-modal and cross-modal) through the same architecture. The first network universally processes spliced multi-modal data, and the second network universally extracts retrieval features, eliminating the need for separate specialized networks for each modality and reducing overall system complexity.
Data Source
AI summary
The present disclosure relates to a multi-modal data retrieval method and apparatus, a medium, and an electronic device. The method includes: inputting target retrieval data into a first feature extraction network corresponding to a modality of the target retrieval data to acquire a data feature of the target retrieval data; inputting the data feature into a second feature extraction network corresponding to the modality of the target retrieval data to acquire a target retrieval feature corresponding to the target retrieval data, wherein second feature extraction networks respectively corresponding to modalities share a weight; and performing retrieval based on the target retrieval feature.