Multi-Modal Feature Extraction Model Training via Cross-Modal Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for determining semantic correlation between multi-modal data, such as text and images, face challenges in improving the accuracy of prediction results from feature extraction models.
Innovation Solution
A model training method that involves obtaining and masking multi-modal data sets, performing feature prediction using a feature extraction model to obtain global recovery and global features, and optimizing the model based on these features to enhance the retrieval of corresponding multi-modal data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If feature extraction model is used to determine semantic correlation between multi-modal data, then prediction can be made, but accuracy of prediction result is insufficient
Solution Approach 1:
The patent segments the feature extraction process into multiple specialized components: a first feature extraction network for text data and a second feature extraction network for image data. Each network is optimized for its specific modal type, extracting features at different granularities (word level, sentence level, image region level) to improve overall prediction accuracy while maintaining reliable semantic correlation determination.
Solution Approach 2:
The patent introduces a cross-modal attention mechanism that operates in a new dimensional space, allowing the model to capture relationships between text and image features that are not apparent in their original feature spaces. This attention mechanism computes attention weights across different modalities, enabling the model to focus on relevant cross-modal associations and improve prediction accuracy.
2Measurement precision
If cross-modal attention mechanism is introduced to improve accuracy, then semantic correlation is enhanced, but computational complexity increases
Solution Approach 1:
The cross-modal attention mechanism is applied selectively at specific locations in the network architecture rather than uniformly across all layers. The attention computation is focused on key interaction points between text and image features, maintaining high prediction accuracy while reducing unnecessary computational complexity in other parts of the model.
Solution Approach 2:
The attention mechanism is segmented into distinct computational modules: text feature encoding, image feature encoding, and cross-modal attention computation. This modular segmentation allows for optimized computation at each stage and facilitates efficient implementation that balances accuracy improvement with computational complexity management.
Data Source
AI summary
A model training method includes obtaining first and second modal data sets including pieces of data, wherein the pieces of data include sub-modal data pieces, and wherein pieces of the first modal data correspond to pieces of the second modal data; obtaining masked data sets by masking a piece of the first and second sub-modal data, respectively; performing feature prediction on the first masked data set and the second modal data set and the second masked data set and the first modal data set, based on ta feature extraction model, to obtain first and second global features and first and second global recovery features; and generating a trained feature extraction model by optimizing the feature extraction model based on the first global recovery features, the first global features, the second global recovery features, and the second global features.