Speech Enhancement Training for Artifact-Resistant Live Captioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech enhancement systems introduce artifacts that degrade the performance of live captioning, and training with simulated data limits the model's adaptability to real-world noise conditions.
Innovation Solution
A multi-task training framework is employed to optimize speech enhancement models by using a combination of simulated and real-world noisy recordings, along with ground truth transcription labels, to generate noise-suppressed speech outputs that retain signal quality and improve live captioning performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If conventional speech enhancement systems are used to remove background noise, then noise suppression is improved, but artifacts are introduced that degrade live captioning performance
Solution Approach 1:
The patent introduces an intermediary loss function (ASR loss) that mediates between noise suppression and artifact reduction. The ASR loss acts as a bridge, guiding the speech enhancement model to produce outputs that are not only clean but also suitable for downstream automatic speech recognition tasks, thereby reducing artifacts while maintaining noise suppression capability
Solution Approach 2:
The speech enhancement model is designed to serve multiple functions simultaneously: it performs traditional noise suppression while also optimizing for live captioning performance. The multi-task training framework enables the model to fulfill both roles, making it universally applicable to different downstream tasks without requiring separate models
2Productivity
If simulated data is used to train speech enhancement models, then training efficiency is improved, but the model's adaptability to real-world noise conditions deteriorates
Solution Approach 1:
The training process uses simulated data as a preliminary step to establish a baseline model with good noise suppression capability. This preliminary training on controlled simulated data is followed by fine-tuning on real-world data, allowing the model to first learn fundamental noise removal skills efficiently, then adapt to real-world variations
Solution Approach 2:
The patent changes the training data parameters from purely simulated to a combination of simulated and real-world data. By adjusting the data source parameter and incorporating real recordings with ground truth transcriptions, the model adapts its parameters to handle the complexity and variability of real-world noise conditions while retaining the efficiency benefits of simulated data training
3Object-affected harmful factors
If noise suppression is applied to audio, then background noise is reduced, but transcription accuracy deteriorates
Solution Approach 1:
The patent implements feedback by using the ASR model to evaluate the output of the speech enhancement model. The ASR loss computed from transcribing the enhanced audio provides feedback to the enhancement model, guiding it to adjust its parameters to produce outputs that improve transcription accuracy while maintaining noise suppression
Solution Approach 2:
The patent merges two previously separate processes into a unified training framework: speech enhancement and automatic speech recognition. By combining these tasks and training them jointly with shared loss functions, the system ensures that noise suppression operations are coordinated with transcription requirements, preventing the degradation of transcription accuracy
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are provided for generating and operating a speech enhancement model optimized for generating noise-suppressed speech outputs for improved human listening and live captioning. A computing system obtains a speech enhancement model trained on a first training dataset to generate noise-suppressed speech outputs and an automatic speech recognition model trained on a second training dataset to generate transcription labels for spoken language utterances. A third training dataset comprising a set of spoken language utterances is applied to the speech enhancement model to obtain a first noise-suppressed speech output which is applied to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances. Speech enhancement model parameters are updated to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs based on a comparison of the noise-suppressed transcription output and ground truth transcription labels.