Multi-Modal Emotion Recognizer Using Segmentation and Dynamics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing emotion recognition systems often output a single emotion value, which is insufficient for providing accurate and varied emotion-based services, and are influenced by initial weight settings, limiting their ability to differentiate emotions across multiple input data sources.
Innovation Solution
A robot system equipped with an emotion recognizer that uses a multi-modal approach, incorporating uni-modal and multi-modal recognizers, including text, speech, and image analysis, to recognize emotions through deep learning models, allowing for the integration of multiple data sources and providing a comprehensive emotion recognition result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single emotion value is output from the analyzer module, then the system complexity is reduced, but the measurement precision and information completeness of emotion recognition deteriorates
Solution Approach 1:
The patent segments the emotion recognition output into multiple independent emotion values (e.g., joy, sadness, anger, fear, surprise, disgust) instead of selecting a single emotion. Each emotion is recognized and output separately with its own confidence score, allowing the system to maintain low complexity while achieving high precision through parallel independent recognition channels.
Solution Approach 2:
The patent adds a new dimension to the output by including both emotion type and confidence score for multiple emotions simultaneously. This transforms the output from a single-dimensional emotion label to a multi-dimensional structure containing emotion category, intensity, and reliability metrics, thereby improving information completeness without significantly increasing system complexity.
2Measurement precision
If multiple data sources are integrated for emotion recognition, then the measurement precision improves, but the device complexity increases due to multiple recognizers and data integration mechanisms
Solution Approach 1:
The patent merges multiple uni-modal recognizers (text, speech, image) into a unified multi-modal recognition framework. The recognizers process different data sources independently and their results are combined through a fusion mechanism that outputs integrated emotion values, achieving high precision through comprehensive data integration while managing complexity through modular architecture.
Solution Approach 2:
The patent creates a universal emotion recognition system that handles multiple data sources (text, speech, image) through a common multi-modal framework. The same recognition architecture processes different input types, allowing the system to achieve high precision across diverse data sources without requiring separate specialized systems for each modality.
3Ease of operation
If initial weight settings are used for each data source, then the ease of operation is improved, but the adaptability deteriorates as the system cannot differentiate emotions across various input data effectively
Solution Approach 1:
The patent implements dynamic weight adjustment for different data sources based on the specific recognition context and input data quality. The system automatically adapts the influence of each modality (text, speech, image) according to the situation, allowing it to effectively differentiate emotions across various input data types while maintaining ease of operation through automatic adaptation rather than manual configuration.
Solution Approach 2:
The patent changes the parameters of data source weighting dynamically based on the recognition task and input characteristics. By adjusting the weight parameters of different modalities according to the specific situation, the system achieves high adaptability in emotion differentiation while maintaining simple operation through automated parameter adjustment rather than fixed initial weights.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
An emotion recognizer according to an aspect of the present invention includes: an uni-modal preprocessor configured to include a plurality of recognizers for each modal learned to recognize emotion information of a user contained in uni-modal input data; and a multi-modal recognizer configured to merge output data of the plurality of recognizers for each modal, and be learned to recognize the emotion information of the user contained in the merged data, wherein the emotion recognizer outputs a complex emotion recognition result including an emotion recognition result of each of the plurality of recognizers for each modal and an emotion recognition result of the multi-modal recognizer.