Multimodal Conversation Emotion Forecasting with Lightweight Tree Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing emotion forecasting methods in conversations face challenges due to low accuracy and high complexity, limiting their deployment in practical applications, particularly in real-time scenarios.
Innovation Solution
A tree-based machine learning model is employed to predict emotions in conversations by capturing inter-speaker dependencies, utilizing speaker diarization, audio and text embedding features, and emotion attributes, with a lightweight architecture suitable for edge devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional emotion forecasting methods are used, then prediction accuracy may be improved, but device complexity and computational requirements increase significantly
Solution Approach 1:
The patent segments the emotion forecasting task into distinct components: audio feature extraction, text feature extraction, speaker embedding generation, and emotion prediction. By processing different modalities separately and combining their representations, the system achieves comprehensive emotion prediction while keeping each component relatively simple and manageable.
Solution Approach 2:
The patent introduces embedding representations as intermediary structures that bridge raw audio/text inputs and emotion predictions. Speaker embeddings and emotion embeddings serve as intermediate features that capture essential characteristics without requiring complex direct mappings, thus reducing overall system complexity while maintaining prediction accuracy.
2Measurement precision
If complex emotion forecasting models are deployed, then prediction accuracy improves, but computational requirements and memory usage increase
Solution Approach 1:
The patent extracts only the most essential features from audio and text inputs using embedding layers, rather than processing complete raw signals. By taking out and retaining only the critical emotional and speaker characteristics in compressed embedding representations, the system reduces computational load while preserving prediction accuracy.
Solution Approach 2:
The patent transforms high-dimensional raw audio and text data into lower-dimensional embedding spaces through parameter transformation. This dimensional reduction changes the parameter representation from complex raw signals to compact numerical vectors, significantly reducing computational requirements for subsequent emotion prediction operations.
3Measurement precision
If high-accuracy emotion prediction models are used, then prediction quality improves, but deployment on edge devices becomes difficult
Solution Approach 1:
The patent uses embedding representations as simplified copies of the essential characteristics of audio and text inputs. These embedding copies capture the core emotional and speaker information in a compact form that can be processed efficiently on resource-constrained edge devices while maintaining prediction quality comparable to more complex models.
Solution Approach 2:
The patent employs dynamic embedding representations that can adapt to different speakers and emotional contexts. The speaker embedding and emotion embedding are dynamically generated based on the specific input characteristics, allowing the model to maintain high accuracy across diverse scenarios while keeping the computational structure relatively simple and deployable on edge devices.
Data Source
AI summary
A method and system for emotion recognition and forecasting are disclosed. The method may include obtaining an audio data of a conversation involving a plurality of speakers and identifying a plurality of turns of the conversation from the plurality of utterances. The method may further include extracting audio embedding features from the plurality of turns, obtaining a plurality of text segments associated with the audio data, extracting text embedding features from the plurality of text segments, obtaining and concatenating speaker embedding features associated with the audio data, obtaining and concatenating a plurality of emotion features corresponding to the plurality of turns. The method further comprises executing a tree-based prediction model to predict emotion features of the plurality of speakers for a subsequent turn of the ongoing conversation based on the audio embedding features, text embedding features, the concatenated speaker embedding features, and the concatenated emotion features.