Multi-Mode Emotion Recognition With Self-Attention Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional emotion recognition methods lack robustness and accuracy due to reliance on a single mode of input, such as text or voice, failing to leverage the complementary information from multiple modes and lacking effective self-attention mechanisms for deep feature extraction.
Innovation Solution
A multi-mode emotion recognition method utilizing a self-attention mechanism to extract features from both spectrogram and text, combining a voice feature extraction network with a Transformer encoder and a pre-trained language model like ALBERT to fuse text and voice features through a multi-head attention mechanism for adaptive decision-making.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If single-mode emotion recognition is used, then the system complexity is low, but the accuracy and robustness of emotion classification deteriorates
Solution Approach 1:
The patent combines multiple emotion recognition modes (text, voice, video) into a unified multi-mode recognition system. The system integrates separate feature extraction networks for each mode and fuses their outputs through a fusion module, achieving improved accuracy while managing complexity through modular architecture design.
Solution Approach 2:
The system segments the emotion recognition task into independent mode-specific components (text feature extraction, voice feature extraction, video feature extraction) that can be processed separately and then combined. This segmentation allows each component to be optimized independently while maintaining overall system manageability.
2Reliability
If single-mode emotion recognition is used, then the processing speed is fast, but the robustness of emotion classification deteriorates
Solution Approach 1:
The patent merges multiple independent emotion recognition models (text-based, voice-based, video-based) into a comprehensive multi-mode system. By combining the strengths of different modalities, the system achieves enhanced robustness where the failure or limitation of one mode can be compensated by others.
Solution Approach 2:
The system employs a composite approach by integrating heterogeneous data sources (text, audio, video) into a unified emotion recognition framework. Each modality contributes unique information that complements the others, creating a more robust and reliable classification system analogous to composite materials combining different properties.
3Measurement precision
If feature fusion from multiple modes is implemented, then the emotion recognition accuracy improves, but the computational complexity increases
Solution Approach 1:
The system performs preliminary feature extraction for each mode independently before fusion, allowing parallel processing of text, voice, and video features. This preliminary action enables computational efficiency by preparing modality-specific features in advance rather than processing all modes simultaneously from raw data.
Solution Approach 2:
The patent introduces a fusion module as an intermediary that efficiently combines features from different modes. This intermediary component manages the computational complexity of multi-mode fusion by implementing optimized fusion strategies that balance accuracy improvement with computational resource consumption.
Data Source
AI summary
Disclosed are a multi-mode emotion recognition method, a system, an electronic device, and a storage medium. The method includes obtaining a spectrogram of a voice to be recognized and a corresponding text and inputting the spectrogram and the text into a multi-mode emotion recognition model to obtain an emotion recognition result output by the multi-mode emotion recognition model. The multi-mode emotion recognition model is trained based on a sample spectrogram, and a corresponding sample text, and a sample emotion recognition result, and is configured to extract a feature from the spectrogram and the text by a self-attention mechanism to obtain the voice features and the text feature, fuse the text feature and voice feature to obtain a multi-mode fusion feature, and make an emotion classification decision to obtain an emotion recognition result based on the text feature, the voice feature, and the multi-mode fusion feature.