Joint ASR Model Semantic Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional automatic speech recognition (ASR) systems face challenges in accurately segmenting long-form speech, as silence does not always accurately demarcate complete thoughts, leading to suboptimal recognition accuracy and user experience.
Innovation Solution
A joint segmenting and ASR model is introduced, which includes an encoder and a decoder, utilizing a bidirectional language model to predict semantic segment boundaries based on punctuation, and incorporating a prediction network to generate probability distributions and end-of-segment indications, trained on audio data with ground-truth labels to improve segmentation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional ASR systems use silence-based segmentation for long-form speech, then the system structure remains simple, but segmentation accuracy deteriorates because silence does not always accurately demarcate complete thoughts
Solution Approach 1:
The patent merges the segmentation function with the ASR model by introducing a joint segmenting and ASR model that simultaneously performs speech recognition and semantic segmentation. The decoder generates both speech recognition hypotheses and end-of-segment indications in a unified architecture, eliminating the need for separate segmentation systems and achieving accurate semantic boundary detection through integrated training with ground-truth labels
Solution Approach 2:
The patent introduces a language model teacher as an intermediary to generate ground-truth end-of-segment labels by detecting punctuation in transcribed text. This teacher model serves as a mediator that translates linguistic structure (punctuation) into semantic segment boundaries, which are then used to train the joint model to recognize meaningful segmentation points in speech
2Measurement precision
If ASR systems process long-form speech as continuous audio, then processing complexity remains low, but recognition accuracy deteriorates due to inability to identify semantic boundaries
Solution Approach 1:
The patent divides long-form speech into multiple semantically complete segments by having the decoder generate end-of-segment indications at appropriate boundaries. This segmentation allows the system to process speech in meaningful units rather than as continuous audio, improving recognition accuracy by ensuring each segment contains complete thoughts while maintaining structured processing through the joint model framework
3Productivity
If ASR systems use simple segmentation approaches, then computational resources are conserved, but user experience deteriorates due to suboptimal recognition quality and increased latency
Solution Approach 1:
The patent performs preliminary segmentation by having the decoder generate end-of-segment indications during the speech recognition process. This preliminary identification of semantic boundaries allows the system to prepare segmented outputs in advance, improving both recognition quality and processing efficiency by organizing results into meaningful units before final delivery to the user
Data Source
AI summary
A joint segmenting and ASR model includes an encoder to receive a sequence of acoustic frames and generate, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame. The model also includes a decoder to generate based on the higher order feature representation at each of the plurality of output steps a probability distribution over possible speech recognition hypotheses, and an indication of whether the corresponding output step corresponds to an end of segment (EOS). The model is trained on a set of training samples, each training sample including audio data characterizing multiple segments of long-form speech; and a corresponding transcription of the long-form speech, the corresponding transcription annotated with ground-truth EOS labels obtained via distillation from a language model teacher that receives the corresponding transcription as input and injects the ground-truth EOS labels into the corresponding transcription between semantically complete segments.