Streaming ASR Joint Speech-Text Training for Domain Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based ASR models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when training data is not extensive, and using synthesized speech for unpaired text data to train ASR models results in performance issues due to mismatches and limited variation.
Innovation Solution
A joint speech and text streaming model that trains on unpaired textual utterances and transcribed speech utterances, using a combination of text and audio encoders and decoders to generate probability distributions, with a parameter-free duration model to align text and speech features, allowing for shared latent representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If ASR models are trained on larger training datasets to improve accuracy and reduce overfitting, then the model's ability to generalize unseen data improves, but the cost of data collection and processing increases
Solution Approach 1:
The patent introduces text data as an intermediary resource to bridge the gap between limited speech data and the need for large-scale training. By processing unpaired text through text encoders and aligning it with speech encoders in a joint training framework, the system leverages abundant text data to improve ASR generalization without requiring proportional increases in labeled speech data
Solution Approach 2:
The joint speech-text encoder is designed to process both speech and text inputs through a unified architecture. The encoder learns shared latent representations that are transferable across modalities, allowing the same model parameters to serve both speech recognition and text understanding tasks, thereby improving generalization with diverse training data
2Quantity of substance
If synthesized speech is used to increase training data volume, then the amount of training data increases, but the variability and realism of the training data decreases
Solution Approach 1:
The patent uses text data as an intermediary to bridge the gap between synthesized speech limitations and real speech variability. By processing unpaired text through text encoders and aligning it with speech encoders, the system captures linguistic variability from text while using it to condition speech representations, thereby improving generalization without relying solely on synthesized speech
Solution Approach 2:
The training approach combines multiple data sources (synthesized speech, unpaired text, and potentially real speech) into a composite training framework. The joint encoder processes heterogeneous inputs from different modalities and sources, learning robust representations that benefit from the diversity of text data while incorporating speech characteristics, creating a more versatile model
3Adaptability or versatility
If unpaired text data is incorporated into ASR training to improve generalization, then the model's adaptability improves, but the complexity of combining speech and text modalities increases
Solution Approach 1:
The patent merges speech encoding and text encoding into a unified joint encoder architecture. The speech encoder and text encoder are combined such that they share parameters and process their respective inputs through a common latent space, enabling the model to learn shared representations that improve generalization while maintaining a relatively compact architecture
Solution Approach 2:
The joint encoder is designed as a universal component that can process both speech and text inputs. By using the same encoder parameters for both modalities and aligning their latent representations, the system achieves multi-functionality without requiring separate specialized models, thereby managing complexity while improving adaptability
4Loss of time
If streaming processing is used to reduce latency in ASR, then the response time improves, but the accuracy may be compromised due to limited context
Solution Approach 1:
The joint speech-text encoder performs preliminary processing of both speech and text inputs in parallel, extracting features and aligning representations before the decoding stage. This preliminary alignment of modalities allows the streaming decoder to make more informed predictions with limited context, improving accuracy without sacrificing latency benefits
Solution Approach 2:
The patent adds a text modality dimension to the traditional speech-only streaming ASR pipeline. By incorporating text embeddings and processing them alongside speech features in the joint encoder, the system enriches the feature space available to the streaming decoder, enabling more accurate predictions with the same latency constraints
Data Source
AI summary
A method includes receiving training data that includes a set of unspoken textual utterances. For each respective unspoken textual utterance, the method includes, tokenizing the respective textual utterance into a sequence of sub-word units, generating a first higher order textual feature representation for a corresponding sub-word unit tokenized from the respective unspoken textual utterance, receiving the first higher order textual feature representation generated by a text encoder, and generating a first probability distribution over possible text units. The method also includes training an encoder based on the first probability distribution over possible text units generated by a first-pass decoder for each respective unspoken textual utterance in the set of unspoken textual utterances.