Joint Neural Network Training for Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional speech recognition technologies face challenges in complex environments with non-stationary noise and multi-speaker interference, leading to degraded performance due to signal errors and signal distortions introduced during the speech separation stage, which are not effectively addressed by independent training of speech separation and phoneme recognition models.
Innovation Solution
A joint training method for neural networks is proposed, where a first subnetwork extracts a target speech spectrum, a second subnetwork adaptively transforms it to obtain an intermediate transition representation, and a third subnetwork performs phoneme recognition, with parameter updates across all subnetworks to minimize recognition loss, thereby connecting the speech separation and phoneme recognition stages and reducing errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If speech separation and phoneme recognition models are trained independently, then each model can be optimized separately, but recognition errors increase due to signal errors and distortions not being addressed across stages
Solution Approach 1:
The patent merges speech separation and phoneme recognition into a unified joint training framework where both models are trained simultaneously with shared parameters. The loss function combines speech separation loss and phoneme recognition loss, enabling gradient backpropagation across both stages to address signal errors end-to-end, thereby improving recognition accuracy while maintaining training feasibility through unified optimization.
2Measurement precision
If a joint training method is used to connect speech separation and phoneme recognition stages, then recognition errors are reduced, but system complexity increases
Solution Approach 1:
The patent segments the joint training system into distinct but connected modules: a speech separation subnetwork, a phoneme recognition subnetwork, and a unified loss function comprising separate loss components. This modular segmentation allows each component to be designed and optimized independently while maintaining joint training benefits, thereby reducing overall system complexity through structured organization.
Solution Approach 2:
The patent introduces an intermediate representation layer between speech separation and phoneme recognition that serves as a bridge. This intermediate layer contains adaptive parameters that are jointly optimized with both stages, acting as a mediator that enables gradient flow and parameter sharing while maintaining clear functional boundaries, thus managing system complexity through controlled coupling.
3Reliability
If deep learning technology is applied to speech recognition, then performance improves in general conditions, but performance degrades in complex environments with non-stationary noise and multi-speaker interference
Solution Approach 1:
The patent implements feedback mechanisms where phoneme recognition errors propagate back to the speech separation stage through joint gradient descent. The unified loss function provides feedback signals that guide the speech separation model to minimize not only separation errors but also downstream recognition errors, enabling the system to adapt to complex acoustic scenarios through continuous error correction across stages.
Solution Approach 2:
The patent employs parameter sharing and adaptive parameter optimization across speech separation and phoneme recognition stages. By jointly optimizing parameters in both stages with a unified loss function, the system dynamically adjusts parameters to handle varying acoustic conditions, thereby improving robustness against non-stationary noise and multi-speaker interference while maintaining good performance in general conditions.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present application relates to the technical field of artificial intelligence, and provides a speech recognition method and apparatus, and a neural network training method and apparatus. The neural network training method is executed by an electronic device, and comprises: obtaining sample data, the sample data comprising a mixed speech spectrum and labeled phonemes thereof; extracting a target speech spectrum from the mixed speech spectrum by a first sub-network; performing adaptive conversion on the target speech spectrum by a second sub-network to obtain an intermediate transition feature; performing phoneme recognition by a third sub-network on the basis of the intermediate transition feature; and updating parameters of the first sub-network, the second sub-network, and the third sub-network according to the phoneme recognition result and the labeled phonemes.