Adversarial Speaker Adaptation for Overfitting-Resistant Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network (DNN)-based acoustic models face suboptimal speech recognition performance due to acoustically mismatched training and runtime input data, and speaker-based adaptation with limited data leads to overfitting.
Innovation Solution
An adversarial speaker adaptation scheme using a discriminator network regularizes the distribution of deep hidden features in a speaker-dependent (SD) DNN acoustic model to match that of a pre-trained speaker-independent (SI) model, optimizing the SD model with a minimax objective to maintain similarity and improve senone discrimination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If speaker-based model adaptation is performed to improve speech recognition performance for a target speaker, then recognition accuracy improves, but the model becomes overfitted to limited adaptation data
Solution Approach 1:
The patent changes the optimization parameters from purely speaker-specific adaptation to a constrained optimization that maintains the distribution characteristics of the original SI model. By modifying the adaptation criterion to include distribution preservation constraints, the model adapts to the target speaker while preventing overfitting through parameter regularization.
Solution Approach 2:
The patent introduces distribution matching as an intermediary constraint between the SI model and the adapted SD model. This intermediary mechanism ensures that while the model adapts to speaker-specific characteristics, it maintains the fundamental distribution properties of the original model, acting as a bridge that prevents extreme overfitting.
2Adaptability or versatility
If the SI DNN acoustic model is adapted to a target speaker using limited adaptation data, then speaker-specific performance improves, but the distribution of the adapted model deviates from the original SI model
Solution Approach 1:
The patent modifies the adaptation parameters by incorporating distribution preservation constraints into the optimization objective. This changes the adaptation process from unconstrained speaker-specific tuning to a constrained optimization that simultaneously maintains model distribution stability and achieves speaker adaptation.
Solution Approach 2:
The patent applies distribution matching as a preventive constraint before the adaptation process fully unfolds. By establishing distribution preservation as a prior constraint, the method cushions against potential distribution drift that would occur during speaker adaptation, ensuring stability is maintained throughout the adaptation process.
3Productivity
If conventional backpropagation is used for model adaptation, then the model learns from adaptation data, but the adaptation criterion does not preserve the original model distribution
Solution Approach 1:
The patent merges the conventional backpropagation learning objective with a distribution preservation constraint. By combining these two objectives into a unified optimization criterion, the method achieves both efficient learning from adaptation data and preservation of the original model distribution simultaneously.
Solution Approach 2:
The patent introduces distribution matching as an intermediary constraint that mediates between the learning objective and the distribution preservation requirement. This intermediary mechanism ensures that while the model efficiently learns from adaptation data through backpropagation, the original distribution characteristics are maintained.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments are associated with a speaker-independent acoustic model capable of classifying senones based on input speech frames and on first parameters of the speaker-independent acoustic model, a speaker-dependent acoustic model capable of classifying senones based on input speech frames and on second parameters of the speaker-dependent acoustic model, and a discriminator capable of receiving data from the speaker-dependent acoustic model and data from the speaker-independent acoustic model and outputting a prediction of whether received data was generated by the speaker-dependent acoustic model based on third parameters. The second parameters are initialized based on the first parameters, the second parameters are trained based on input frames of a target speaker to minimize a senone classification loss associated with the second parameters, a portion of the second parameters are trained based on the input frames of the target speaker to maximize a discrimination loss associated with the discriminator, and the third parameters are trained based on the input frames of the target speaker to minimize the discrimination loss.