Multi-modal Encoder Cross-Modal Fusion via Masked Feature Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mainstream multi-modal fusion learning methods struggle to effectively share parameters between modalities and require manual conversion of end task formats, making it difficult to implement effective fusion of multi-modal features.
Innovation Solution
A multi-modal encoder processing method and apparatus that uses a pair of mask samples, including a text sample and an audio sample, to generate text and audio encoding features by fusing linear spectrum features and linear word features, respectively, allowing for effective sharing and fusion of audio and text features through one training task.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a unified network architecture with self-attention mechanism is used to process multiple modalities, then the processing capability is improved, but parameter sharing between modalities is not effective and manual conversion is required
Solution Approach 1:
The patent merges audio and text processing into a unified encoder architecture where both modalities share the same parameter space. The audio encoder and text encoder use identical transformer block structures with shared attention mechanisms, eliminating the need for separate processing paths and manual format conversions while maintaining effective multi-modal feature fusion
2Reliability
If separate encoders are used for audio and text, then modality-specific features are preserved, but parameter sharing between modalities is not achieved
Solution Approach 1:
The patent implements universal encoder blocks that can process both audio and text inputs using the same parameters. The transformer blocks are designed to be modality-agnostic, accepting embeddings from either audio or text inputs and producing unified representations, thereby achieving parameter sharing while maintaining reliability through consistent processing
3Adaptability or versatility
If double-tower model with comparison learning is used, then multi-modal fusion is achieved, but effective parameter sharing between modalities is not realized
Solution Approach 1:
The patent merges the separate audio and text encoders into a single unified encoder structure where both modalities are processed through shared transformer blocks. This consolidation achieves multi-modal fusion while maximizing parameter sharing efficiency, as the same weights are used for both audio and text processing without requiring separate tower structures
Data Source
AI summary
The present disclosure provides a multi-modal encoder processing method and apparatus, a computer device and a storage medium. The method includes: acquiring a pair of mask samples to be processed, the pair of mask samples including a text sample and an audio sample associated with each other, and at least one of the text sample and the audio sample is masked; based on a multi-modal encoder, generating a text encoding feature of the text sample, and generating an audio encoding feature of the audio sample, a linear spectrum feature of the audio sample being fused in the text encoding feature, and a linear word feature of the text sample being fused in the audio encoding feature; and predicting masked mask information according to the text encoding feature and the audio encoding feature, and correcting the multi-modal encoder based on an accuracy of the mask information.