Joint Speech Recognition Training for Noisy Multi-Speaker Audio
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech recognition technologies suffer from low accuracy in challenging conditions, such as interference from other speakers or music, due to the lack of effective integration of speech separation, enhancement, and recognition models.
Innovation Solution
A joint training method is employed to fuse loss functions from speech separation and enhancement models with a robust representation model and a speech recognition model, using intermediate neural networks to bridge the gap between human-oriented and machine-oriented speech processing tasks, enabling end-to-end training and improving recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speech separation and enhancement are performed separately from speech recognition, then the processing pipeline is simpler and more modular, but the speech recognition accuracy is low in challenging conditions
Solution Approach 1:
The patent merges speech separation, speech enhancement, and speech recognition models into a unified end-to-end trainable system. The separated models are integrated such that they share common representations and can be jointly optimized through a single loss function, thereby improving recognition accuracy while maintaining reasonable system complexity through shared computational layers.
Solution Approach 2:
The patent creates a multi-functional system where the speech processing model serves multiple purposes: it performs speech separation, speech enhancement, and speech recognition simultaneously. The shared neural network layers extract features that are useful for all three tasks, making the system more efficient and accurate compared to separate specialized models.
2Reliability
If conventional separate models are used for speech separation and recognition, then the system is easier to implement, but the word error rate is high under noise conditions
Solution Approach 1:
The patent combines multiple speech processing functions into a single integrated model that can be implemented end-to-end. This merging approach improves noise robustness by allowing the model to learn correlations between separation, enhancement, and recognition tasks, while the unified implementation simplifies deployment compared to coordinating multiple separate models.
Solution Approach 2:
The patent introduces intermediate representations and shared feature extraction layers that act as mediators between the input speech signal and the final recognition output. These intermediate layers enable the model to progressively refine the speech representation, improving robustness to noise while maintaining a straightforward sequential implementation structure.
3Measurement precision
If speech separation and enhancement are performed first, then the speech signal is cleaner, but the overall system accuracy remains limited without joint optimization
Solution Approach 1:
The patent ensures continuous optimization throughout the entire speech processing pipeline by enabling end-to-end training. The loss function propagates gradients continuously from the recognition output back through the enhancement and separation stages, allowing the model to continuously refine all intermediate representations to maximize final recognition accuracy rather than optimizing each stage independently.
Solution Approach 2:
The patent implements feedback mechanisms through backpropagation where the recognition loss is fed back to adjust the parameters of the speech separation and enhancement components. This feedback loop allows the model to learn how different separation and enhancement strategies affect final recognition accuracy, thereby optimizing the entire pipeline for the ultimate goal of accurate speech recognition.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Provided are a voice recognition method, device, and computer-readable storage medium, said method comprising: obtaining a first loss function of a voice separation enhancement model and a second loss function of a voice recognition model (S202); performing back-propagation on the basis of the second loss function to train an intermediate model bridged between the voice separation enhancement model and the voice recognition model, to obtain a robust representation model (S204); combining the first loss function and the second loss function to obtain a target loss function (S206); performing joint training of the voice separation enhancement model, the robust representation model, and the voice recognition model on the basis of the target loss function, and ending training when a preset convergence condition is satisfied (S208).