Speech Emotion Retrieval Using LLM-Generated Captions and CLAP
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech emotion recognition (SER) systems face challenges in data collection due to high costs and time consumption, and existing classification models lack diversity in retrieving out-of-domain (OOD) samples and finer emotion expressions, leading to limited retrieval performance and increased computational requirements.
Innovation Solution
A contrastive language-audio pre-training (CLAP) model, named CLAP4Emo, uses a large language model like ChatGPT to generate emotion captions from a lexicon, paired with audio segments to create audio-text pairs for training, employing a supervised random pairing strategy to enhance model flexibility and reduce computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If mainstream machine learning models are trained with predefined category labels, then the model can predict specific categories accurately, but the model lacks flexibility to predict unseen classes or out-of-domain acoustic conditions
Solution Approach 1:
The patent segments the emotion recognition task into multiple granular emotion categories (e.g., joy, sadness, anger, fear, disgust, surprise) rather than using broad predefined labels. This segmentation allows the model to learn finer-grained representations that can be combined to predict both known and unseen emotion classes, resolving the contradiction between specific category accuracy and flexibility for unseen classes.
Solution Approach 2:
The patent introduces a new dimension by using contrastive learning to create embedding spaces for both audio and text modalities. This dimensional transformation allows the model to operate in a continuous vector space where similarities can be computed for any emotion description, enabling prediction of unseen classes while maintaining accuracy for known categories through the structured embedding space.
2Adaptability or versatility
If more audio segments and emotion categories are collected to improve model diversity, then the model can retrieve out-of-domain samples better, but the data collection cost and time consumption increase significantly
Solution Approach 1:
The patent employs self-supervised contrastive learning where the model learns from the data itself without requiring extensive manual annotation. The system uses automatic text-to-speech generated captions and metadata from existing audio databases to create training pairs, allowing the model to serve itself by learning representations from unlabelled or minimally labelled data, thus reducing data collection time while maintaining diversity.
Solution Approach 2:
The patent uses text-to-speech synthesis to generate synthetic audio captions that copy the emotional content of reference audio segments. By creating synthetic training examples that replicate emotional patterns, the system expands its training data diversity without requiring proportional increases in real recorded data collection, thereby reducing time and resource costs.
3Device complexity
If traditional classification models are used for speech emotion recognition, then the model structure is simple, but the computational requirements increase when handling diverse emotion expressions and out-of-domain samples
Solution Approach 1:
The patent replaces traditional classification-based mechanical systems with a contrastive learning approach that uses embedding spaces and similarity computations. This substitution allows the model to handle diverse emotion expressions more efficiently by computing cosine similarities in a fixed-dimensional space rather than performing complex classification operations, reducing computational requirements while maintaining structural simplicity.
4Ease of manufacture
If predefined category labels are used for training, then the training process is straightforward, but the model cannot effectively retrieve fine-grained emotion expressions
Solution Approach 1:
The patent applies local quality by creating specialized embedding spaces for different emotion dimensions and using fine-grained emotion categories (e.g., distinguishing between different types of joy or sadness). This allows the model to maintain simple overall training procedures while achieving high precision in retrieving specific fine-grained emotion expressions through localized optimization in different regions of the embedding space.
Data Source
AI summary
Methods and systems for generating training data for training a contrastive language-audio machine-learning model. A plurality of audio segments are retrieved from a speech emotion recognition (SER) database along with metadata associated with the audio segments. The metadata of each audio segment includes an emotion class. Words or terms associated with emotions are retrieved from a lexicon. A large language model (LLM) is executed on (i) the classes of emotion associated with the audio segments and (ii) the words or terms from the lexicon. This generates a plurality of text captions associated with emotion, which are stored in a caption pool. For each audio segment retrieved from the SER database, that audio segment is paired with one or more of the text captions from the caption pool that were generated based on the emotion class associated with that audio segment. This yields audio-text pairs for training a contrastive learning model.