Decoupled Attention Transformer for Unified Multi-Modal Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-modal data pre-training models are limited by their reliance on strongly-correlated image-text pair data, leading to deficiencies in generality, inability to uniformly model different modal data, and difficulty in performing on single-modal tasks, with limited data scale and practical applicability.
Innovation Solution
A method for training a model using multi-modal data joint learning, employing a decoupling attention Transformer network model that processes single-modal and Pair multi-modal data to generate semantic representation features, utilizing intra-modal and inter-modal attention mechanisms, and adaptive information fusion to enhance robustness and generality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-modal data pre-training models use strongly-correlated image-text pair data, then cross-modal semantic representation is improved, but generality and adaptability to single-modal tasks deteriorate
Solution Approach 1:
The patent segments the attention mechanism into intra-modal attention (processing single-modal data independently) and inter-modal attention (processing multi-modal data jointly). This segmentation allows the model to handle strongly-correlated image-text pairs with inter-modal attention while maintaining generality through intra-modal attention on single-modal data, resolving the contradiction between cross-modal accuracy and adaptability
Solution Approach 2:
The decoupling attention Transformer network model is designed with universal applicability to handle both single-modal and multi-modal data through a unified architecture. The model can process image data, text data, or paired data using the same base structure with different attention mechanisms activated, enabling it to perform well on both cross-modal and single-modal tasks
2Measurement precision
If task-oriented division focuses on single-modal data, then single-modal understanding is improved, but multi-modal data learning capability deteriorates
Solution Approach 1:
The patent merges single-modal processing and multi-modal processing into a unified decoupling attention Transformer network model. By combining intra-modal attention mechanisms for single-modal data with inter-modal attention mechanisms for multi-modal data, the model achieves both specialized single-modal understanding and integrated multi-modal learning capability
3Measurement precision
If existing models are designed for specific modalities, then modality-specific performance is improved, but unified data processing capability deteriorates
Solution Approach 1:
The patent employs a universal decoupling attention Transformer network model that can process image data, text data, and paired data through the same architecture. The model uses configurable attention mechanisms (intra-modal or inter-modal) depending on the input data type, achieving unified data processing without requiring separate specialized models for each modality
Data Source
AI summary
A method for training a model based on multi-modal data joint learning, includes: obtaining multi-modal data; in which the multi-modal data include at least one type of single-modal data and at least one type of Pair multi-modal data; inputting the single-modal data and the Pair multi-modal data into a decoupling attention Transformer network model to generate respectively Token semantic representation features and cross-modal semantic representation features; and training the decoupling attention Transformer network model based on the Token semantic representation features and the cross-modal semantic representation features.