Speech Emotion Recognition Using LLM Textual Entailment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech emotion recognition (SER) methods face challenges in in-the-wild scenarios due to variability in natural speech and reliance on human ratings with limited emotion taxonomies, making it costly and resource-intensive to extend model training to large, natural speech datasets.
Innovation Solution
The technology employs a textual entailment approach using pre-trained large language models to infer weak emotion labels from unlabeled speech data, enabling weakly-supervised learning of SER models. This involves generating a text transcript from input speech, applying it to a pre-trained language model with an engineered prompt and predetermined taxonomy, and generating a predicted emotion through textual entailment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human ratings are used to label speech data for emotion recognition, then label accuracy is improved, but computational cost and time resources increase significantly
Solution Approach 1:
The patent uses pre-trained large language models to generate synthetic emotion labels that copy the functionality of human-rated labels without requiring actual human annotation. The LLM generates predicted emotion labels from speech transcripts, creating artificial labels that replace the need for expensive human ratings while maintaining sufficient accuracy for training SER models
Solution Approach 2:
The system enables self-service labeling where the language model automatically generates emotion labels from speech transcripts without human intervention. The LLM processes transcripts and generates predicted emotion labels independently, allowing the system to label large datasets without requiring human raters for each sample
2Adaptability or versatility
If large, natural speech datasets are used for model training, then model generalization is improved, but resource requirements and complexity increase
Solution Approach 1:
The patent introduces an intermediary process using pre-trained language models to bridge between raw speech data and SER model training. The LLM acts as a mediator that converts speech transcripts into predicted emotion labels, enabling the use of large natural speech datasets without requiring proportional increases in annotation resources or system complexity
Solution Approach 2:
The system performs preliminary action by pre-training language models on large corpora before using them for emotion label generation. The pre-trained LLM already contains linguistic knowledge and emotion understanding from prior training, allowing it to generate accurate emotion labels without requiring the SER system itself to be complex or resource-intensive
3Ease of operation
If limited emotion taxonomies are used, then labeling process is simplified, but emotion recognition accuracy deteriorates
Solution Approach 1:
The patent changes the parameter of emotion taxonomy granularity by using pre-trained language models that can output detailed emotion categories. The LLM can generate fine-grained emotion labels (such as specific emotions like joy, sadness, anger, fear) automatically, allowing the system to use comprehensive taxonomies without increasing manual labeling complexity since the LLM handles the detailed classification
Data Source
AI summary
The technology relates to enhancing speech emotion recognition models with methods that enable the use of unlabeled data by inferring weak emotion labels. This is done by pre-trained large language models through weakly-supervised learning. For inferring weak labels constrained to a taxonomy, a textual entailment approach selects an emotion label with the highest entailment score for a speech transcript extracted via automatic speech recognition. The system may employ a method that generates, by one or more processors, a text transcript for a snippet of input speech, and then applies the text transcript to a pre-trained language model. The system can generate, using the pre-trained language model according to an engineered prompt and a predetermined taxonomy, a textual entailment from the text transcript. Based on this, the system may generate, by the one or more processors using the textual entailment, a predicted emotion corresponding to the input speech.