Joint Speech-Text Streaming ASR With Shared Encoder Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based ASR models tend to overfit training data, leading to difficulties in generalizing unseen data, especially when the training data is not extensive, and using synthesized speech for unpaired text data to train ASR models results in inaccuracies due to mismatches in speech variation and disfluencies, impacting performance in streaming applications.
Innovation Solution
A joint speech and text streaming model that trains an encoder using unpaired textual utterances and transcribed speech, employing a semi-supervised training process with a parameter-free alignment model to generate alignment outputs, and a combination of causal and non-causal loss terms to improve generalization across domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate speech and text streaming models are used, then model specialization is achieved, but system complexity increases and inference time doubles
Solution Approach 1:
The patent combines separate speech and text streaming models into a single unified model that processes both audio and text inputs simultaneously. This merging eliminates the need for sequential inference through two separate models, reducing system complexity while maintaining specialized capabilities for both speech and text processing through shared transformer architecture and multi-modal input handling.
Solution Approach 2:
The unified model is designed with multi-functionality to handle both speech-to-text and text-to-text processing tasks within a single architecture. The model accepts multiple input types (audio features and text tokens) and produces text outputs, making it universally applicable to different processing scenarios without requiring separate specialized models.
2Reliability
If separate speech and text streaming models are used, then specialized processing is achieved, but total inference time increases
Solution Approach 1:
The unified model enables continuous processing of both speech and text streams simultaneously through its streaming architecture. Instead of sequential processing through separate models, the single model processes audio and text inputs in parallel streams, maintaining continuous useful action and reducing total inference time while preserving specialized processing capabilities.
Solution Approach 2:
The model performs preliminary processing of both audio and text inputs through shared embedding layers and attention mechanisms before final prediction. This preliminary action on multiple input types simultaneously prepares features for downstream processing, reducing the overall inference time compared to sequential processing of separate models.
3Productivity
If joint speech and text processing is implemented, then processing efficiency improves, but model training complexity increases
Solution Approach 1:
The training process is segmented into distinct loss components that can be computed and optimized separately: speech recognition loss, text prediction loss, and auxiliary losses. This segmentation allows complex multi-task training to be broken down into manageable components, each with its own optimization strategy, reducing training complexity while maintaining processing efficiency.
Solution Approach 2:
The model employs parameter sharing across different processing tasks to reduce training complexity. By using shared embedding layers, attention mechanisms, and transformer blocks for both speech and text processing, the number of independent parameters to be trained is reduced, simplifying the training process while enabling efficient joint processing through unified feature representations.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A method (500) includes receiving training data that includes a set of unspoken textual utterances (320). For each respective unspoken textual utterance, the method includes, tokenizing the respective textual utterance into a sequence of sub-word units (402), generating a first higher order textual feature representation (203) for a corresponding sub-word unit tokenized from the respective unspoken textual utterance, receiving the first higher order textual feature representation generated by a text encoder (202), and generating a first probability distribution (253) over possible text units. The method also includes training an encoder (210) based on the first probability distribution over possible text units generated by a first decoder (250) for each respective unspoken textual utterance in the set of unspoken textual utterances.