Audio Encoder Pre-training with Synthetic Speech for ASR Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based ASR models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when the training data is not extensive enough.
Innovation Solution
A computer-implemented method that pre-trains an audio encoder to jointly learn shared representations of speech and text by using training data that includes unspoken text utterances and un-transcribed non-synthetic speech utterances, generating synthetic speech representations, and applying contrastive and supervised losses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If ASR models are trained on limited training data, then training time and data requirements are reduced, but the models overfit and fail to generalize to unseen data
Solution Approach 1:
The patent introduces a text encoder as an intermediary component that processes text representations alongside audio inputs. This text encoder learns shared representations with the audio encoder through contrastive learning, acting as a mediator that helps the model generalize better from limited training data by leveraging text as an auxiliary modality.
Solution Approach 2:
The patent applies pre-training on large-scale unlabeled audio data before fine-tuning on smaller labeled datasets. This preliminary self-supervised learning phase allows the model to learn robust acoustic features and representations that transfer to downstream tasks, improving generalization when final training data is limited.
2Reliability
If ASR models are trained on larger training datasets, then accuracy and generalization improve, but training complexity and computational resources increase
Solution Approach 1:
The patent implements self-supervised learning where the model learns from unlabeled audio data without requiring extensive manual annotation. The contrastive learning objective uses the audio input and its corresponding text representation as self-supervised signals, eliminating the need for large labeled datasets and reducing training complexity.
Solution Approach 2:
The audio encoder is designed to serve multiple functions: it processes both raw audio inputs and synthesized speech representations, and its learned representations are used for both contrastive learning with the text encoder and downstream ASR tasks. This multi-functionality reduces the need for separate models and simplifies the overall training system.
3Quantity of substance
If synthesized speech is used to increase training data volume, then data availability improves, but data quality and realism may deteriorate
Solution Approach 1:
The patent merges synthesized speech representations with real audio inputs in the training process. The audio encoder processes both types of inputs jointly, and the contrastive learning objective ensures that the model learns to distinguish between them while learning shared representations with the text encoder, thereby maintaining data quality while increasing volume.
Data Source
AI summary
A method includes receiving training data that includes unspoken text utterances and un-transcribed non-synthetic speech utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation for each unspoken textual utterance of the received training data using a text-to-speech model. The method also includes pre-training an audio encoder on the synthetic speech representations generated for the unspoken textual utterances and the un-transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.