Long-Form Speech Recognition Training With Context and WER Minimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automatic speech recognition (ASR) systems struggle with long-form utterances due to their conversational nature, presence of non-speech signals, and noisy acoustic conditions, leading to poor performance and generalization issues.
Innovation Solution
Training ASR models on long-form training utterances that include contextual information and non-speech segments, using a transducer-based model with an encoder, prediction, and joint network to minimize word error rate without segmenting incoming speech.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional ASR systems process long-form utterances by segmenting incoming speech, then processing complexity is reduced, but transcription accuracy deteriorates due to loss of contextual information
Solution Approach 1:
The patent applies segmentation by dividing long-form speech into fixed-time segments (e.g., 30-second segments with 50% overlap) for parallel processing. Each segment is processed independently by separate ASR models, reducing the computational burden on individual models while maintaining overall accuracy through contextual preservation within each segment.
Solution Approach 2:
The patent introduces a temporal dimension by processing multiple overlapping time segments simultaneously. Instead of processing the entire long-form utterance as a single unit in one dimension, the system creates multiple overlapping temporal windows that capture contextual information from different time perspectives, thereby maintaining accuracy while enabling parallel processing.
2Quantity of substance
If ASR models are trained only on short utterances, then training data availability is improved, but generalization performance deteriorates on long-form speech
Solution Approach 1:
The patent applies preliminary action by pre-training ASR models on abundant short utterance data to establish strong baseline performance. Subsequently, the models are fine-tuned or adapted to handle long-form speech through the segmented processing approach, allowing the system to leverage both the quantity of short utterance training data and the requirements of long-form transcription.
Solution Approach 2:
The patent creates a universal processing framework that can handle both short and long utterances. The same ASR model architecture and segmentation approach work for various utterance lengths, making the system multi-functional. This allows the system to benefit from training data of any length while maintaining consistent performance across different speech durations.
3Loss of time
If ASR systems process long-form speech in real-time streaming fashion, then latency is reduced, but transcription accuracy deteriorates due to limited contextual information
Solution Approach 1:
The patent applies periodic action by processing speech in regular, fixed-time segments with systematic overlap. Each segment is processed at regular intervals, allowing the system to provide periodic transcription updates in real-time while maintaining consistent contextual analysis through the overlapping segment structure.
Solution Approach 2:
The patent uses preliminary action by pre-processing and segmenting the incoming speech stream into fixed-time windows before ASR processing. This preliminary segmentation prepares the data in advance, allowing the ASR models to process pre-defined segments with their contextual information already captured, thereby reducing processing latency while maintaining accuracy.
Data Source
AI summary
A method includes obtaining a set of training samples, wherein each training sample includes a corresponding sequence of speech segments corresponding to a training utterance and a corresponding sequence of ground-truth transcriptions for the sequence of speech segments, and wherein each ground-truth transcription includes a start time and an end time of a corresponding speech segment. For each training sample in the set of training samples, the method includes processing, using a speech recognition model, the corresponding sequence of speech segments to obtain one or more speech recognition hypotheses for the training utterance; and, for each speech recognition hypothesis obtained for the training utterance, identifying a respective number of word errors relative to the corresponding sequence of ground-truth transcriptions. The method trains the speech recognition model to minimize word error rate based on the respective number of word errors identified for each speech recognition hypothesis obtained for the training utterance.