Audio Speech Sentiment Classifier Using Pseudo Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech sentiment analysis methods lose rich acoustic information and require large, manually annotated datasets, making them inefficient and costly.
Innovation Solution
A system that pretrains an audio speech sentiment classifier using pseudo sentiment labels generated by a text-trained pseudo labeler, allowing for unsupervised pretraining and subsequent fine-tuning with a smaller human-annotated dataset, thereby incorporating acoustic information and reducing annotation needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional ASR-based sentiment analysis is used, then text classification can be performed, but rich acoustic information is lost
Solution Approach 1:
The patent merges the ASR encoder and sentiment classifier into a single end-to-end model that processes audio speech directly. The audio speech input is fed into an audio encoder that creates a vector representation, which is then passed to the sentiment classifier, eliminating the need for separate ASR and text classification steps while preserving acoustic information.
Solution Approach 2:
The patent introduces a pseudo labeler as an intermediary component that generates pseudo sentiment labels from text transcriptions. This pseudo labeler serves as a bridge between the audio encoder and the sentiment classifier, enabling unsupervised pretraining by providing target labels without requiring manual annotation of the entire dataset.
2Loss of information
If end-to-end sentiment analysis is used, then acoustic information is preserved, but large sentiment-annotated datasets are required
Solution Approach 1:
The system performs self-service by using the pseudo labeler to automatically generate pseudo sentiment labels for the training dataset. This eliminates the need for manual annotation of the entire dataset, as the system generates its own training labels through the pseudo labeler, which is pre-trained on sentiment-annotated text data.
Solution Approach 2:
The patent applies preliminary action by pre-training the pseudo labeler on sentiment-annotated text data before using it to generate pseudo labels for the audio speech dataset. This pretraining phase prepares the pseudo labeler to accurately generate sentiment labels, reducing the need for manual annotation in the main task.
3Measurement precision
If manual annotation is used to train sentiment classifiers, then accurate sentiment labels are obtained, but time and cost increase significantly
Solution Approach 1:
The patent uses copying by creating pseudo sentiment labels that replicate the function of manually annotated labels. The pseudo labeler generates these labels by copying the sentiment classification capability from pre-trained text models, allowing the system to train without requiring actual manual annotation while maintaining sufficient accuracy for effective training.
4Ease of manufacture
If text-trained sentiment classifiers are used, then training data is available, but acoustic/prosodic information is not utilized
Solution Approach 1:
The patent applies universality by designing a sentiment classifier that can process both text and audio inputs through a unified architecture. The audio encoder processes speech audio to create vector representations, which are then fed to the same sentiment classifier used for text, allowing the system to utilize acoustic information while maintaining the benefits of pre-trained text models.
Data Source
AI summary
The present disclosure describes a system, method, and computer program for predicting sentiment labels for audio speech utterances using an audio speech sentiment classifier pretrained with pseudo sentiment labels. A speech sentiment classifier for audio speech (“a speech sentiment classifier”) is pretrained in an unsupervised manner by leveraging a pseudo labeler previously trained to predict sentiments for text. Specifically, a text-trained pseudo labeler is used to autogenerate pseudo sentiment labels for the audio speech utterances using transcriptions of the utterances, and the speech sentiment classifier is trained to predict the pseudo sentiment labels given corresponding embeddings of the audio speech utterances. The speech sentiment classifier is then subsequently fine tuned using a sentiment-annotated dataset of audio speech utterances, which may be significantly smaller than the unannotated dataset used in the unsupervised pretraining phase.