Real-Time Emotion Recognition Using Joint Speech-Text Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing emotion recognition technologies primarily rely on data-driven techniques that are successful under controlled environments but fail to effectively recognize emotions in real-life conditions, particularly when emotion changes or new data is introduced, and they do not consider both audio and textual information for comprehensive emotion recognition.
Innovation Solution
A method involving automatic speech recognition, speech representation, emotion prediction, and novelty detection models to compute joint and emotion embedding vectors, allowing for real-time emotion recognition by integrating both audio and textual features, with the ability to adapt to novel expressions through retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data-driven techniques are used for emotion recognition, then emotion detection can be performed under controlled environments, but the system fails to recognize emotions accurately in real-life conditions when emotion changes or new data is introduced
Solution Approach 1:
The system implements dynamic adaptation by continuously learning from new data streams. The emotion recognition model is designed to update its parameters and adapt to changing emotional patterns in real-time, transitioning from static controlled-environment training to dynamic real-life adaptation. This allows the system to maintain reliability while gaining versatility across different contexts.
Solution Approach 2:
The system performs preliminary training on diverse datasets covering multiple emotions and contexts before deployment. This pre-adaptation phase prepares the model to handle real-life variations more effectively, reducing the gap between controlled environment performance and real-world applicability.
2Device complexity
If only audio signals are processed for emotion recognition, then the system is simpler to implement, but the emotion recognition is incomplete as it does not consider both audio and textual information
Solution Approach 1:
The system merges audio signal processing with text analysis by integrating speech-to-text conversion and combining both modalities for joint emotion recognition. This multi-modal approach ensures comprehensive emotion detection by leveraging both acoustic features and linguistic content, preventing information loss while maintaining manageable system complexity through unified architecture.
3Measurement precision
If known emotion recognition techniques are used, then they can successfully recognize emotions under controlled environments, but they do not work well in real life conditions
Solution Approach 1:
The system dynamically adjusts recognition parameters and thresholds based on the operating context. In controlled environments, it uses stricter precision-oriented parameters, while in real-life conditions, it adapts parameters to accommodate greater variability. This parameter flexibility allows the system to maintain measurement precision across different environments while improving real-life applicability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a computer implemented method for a computer implemented method (200) for real-time emotion recognition from a real-time audio signal (102), said method (200) comprising several iterations of an emotion detection phase comprising the following steps: -transcription (106), into text, an audio speech signal contained is said audio signal (102) by an automatic speech recognition model, - computing (112), by a speech representation model, a joint representation vector (JRVi) corresponding to a joint representation of said speech as a function of said speech signal and said text, - computing (114), by an emotion prediction model, an emotion embedding vector (EEVi), as a function of said joint representation vector (JRVi), and - mapping (116) said emotion in at least one emotional frame (120;124), according to said emotion embedding vector (EEVi), by an emotion mapping model. The invention further relates to a computer program and a device implementing such a method.