ASR Audio Encoder Pretraining with Synthetic Speech and Contrastive Losses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based automatic speech recognition (ASR) models tend to overfit training data, leading to difficulties in generalizing to unseen data, especially when the training data is not extensive enough.
Innovation Solution
The method involves pre-training an audio encoder using a combination of contrastive self-supervised losses, supervised losses, and consistency losses, leveraging un-transcribed non-synthetic speech utterances, synthetic speech representations, and transcribed non-synthetic speech utterances to jointly learn speech and text representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ASR models are trained with extensive training data, then accuracy improves, but model complexity and training resources increase
Solution Approach 1:
The patent applies pre-training on synthetic speech data before fine-tuning on real speech data. This preliminary action allows the model to learn general speech patterns and representations from easily generated synthetic data, reducing the amount of complex real training data needed to achieve high accuracy while avoiding excessive model complexity
Solution Approach 2:
The patent uses text-to-speech synthesis to create synthetic speech copies that mimic real speech patterns. These synthetic copies serve as proxies for real speech data, allowing the model to learn from abundant synthetic data without requiring proportionally large amounts of real speech data, thus improving accuracy without linearly increasing training resource requirements
2Productivity
If ASR models are trained with limited training data, then training resources are reduced, but overfitting increases and generalization ability decreases
Solution Approach 1:
The patent generates synthetic speech copies from text using TTS systems, creating augmented training data without requiring additional real speech recordings. This copying approach allows the model to train on diverse synthetic examples that improve generalization ability while maintaining training efficiency with limited real data resources
Solution Approach 2:
The patent modifies speech data parameters by applying various transformations to synthetic speech (noise addition, pitch shifting, time stretching) to create diverse training examples. These parameter changes enable the model to learn robust representations that generalize better to unseen data while training on limited real speech data
3Quantity of substance
If synthesized speech is used to increase training data volume, then data quantity increases, but data quality and realism may decrease
Solution Approach 1:
The patent uses synthetic speech as a preliminary training step before fine-tuning on real speech data. This two-stage approach allows the model to first learn general patterns from abundant synthetic data, then refine its performance on high-quality real data, achieving both large effective training volume and high data quality
Solution Approach 2:
The patent creates a composite training dataset combining synthetic speech and real speech data. This composite approach leverages the advantages of both data types: the abundance and diversity of synthetic data plus the authenticity and quality of real data, achieving both high quantity and high quality in the effective training set
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method (600) includes receiving training data that includes unspoken text utterances (320), un-transcribed non-synthetic speech utterances (306), and transcribed non-synthetic speech utterances (304). Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non- synthetic speech utterance is not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance is paired with a corresponding transcription (302). The method also includes generating a corresponding synthetic speech representation (332) for each unspoken textual utterance of the received training da.ta using a text-to-speech model (330). The method also includes pre-training an audio encoder (210) on the synthetic speech representations generated for the unspoken textual utterances, the un-transcribed non-synthetic speech utterances, and the transcribed non- synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations