End-to-End Speech Recognition Model Training with Policy Gradient Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep end-to-end speech recognition models face a disparity between the objective function used during training and the evaluation criteria during inference, leading to suboptimal performance metrics such as word error rate (WER) and character error rate (CER), as they maximize the likelihood of correct transcriptions while ignoring the relative plausibility of incorrect ones.
Innovation Solution
The implementation of a multi-objective learning criteria that combines a maximum likelihood objective function with a policy gradient function to optimize both the probability of correct transcriptions and the penalization of incorrect transcriptions based on their conformity to ground truth, using reinforcement learning to directly address non-differentiable performance metrics like WER and CER.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If maximum likelihood objective function is used for training, then the probability of correct transcription is maximized, but the relative performance metrics (WER and CER) deteriorate due to ignoring the plausibility of incorrect transcriptions
Solution Approach 1:
The patent combines maximum likelihood training with reinforcement learning policy optimization into a unified training framework. The model simultaneously learns from both the probabilistic likelihood of correct transcriptions and the discrete performance metrics (WER/CER) through policy gradients, merging two previously separate training objectives into one coherent system that addresses both aspects of transcription quality
Solution Approach 2:
The patent introduces policy gradient parameters that modify the model's transcription policy based on performance metric feedback. By changing the optimization parameters from purely likelihood-based to include policy gradients derived from WER/CER metrics, the system adapts to directly optimize for the evaluation criteria that matter in real-world deployment
2Productivity
If maximum likelihood objective function is used, then training converges to maximize log probability of correct transcription, but the evaluation criteria during inference (WER and CER) cannot be directly optimized due to being non-differentiable
Solution Approach 1:
The patent introduces policy gradients as an intermediary mechanism that bridges the gap between differentiable likelihood-based training and non-differentiable performance metric optimization. The policy gradient acts as a mediator that translates discrete WER/CER evaluations into continuous gradient signals that can guide parameter updates during training, enabling indirect optimization of non-differentiable metrics through differentiable approximations
Solution Approach 2:
The patent makes the training objective dynamic by switching between and combining two different optimization approaches: maximum likelihood for stable convergence and policy gradient for metric-specific optimization. The training process dynamically adapts by incorporating policy gradient updates that respond to performance metric feedback, allowing the system to adjust its optimization strategy based on the specific evaluation criteria being targeted
3Reliability
If MLOF maximizes log probability of correct transcription, then all incorrect transcriptions are equally penalized, but performance metrics like WER penalize incorrect transcriptions differently based on their edit distance to ground truth
Solution Approach 1:
The patent applies local quality by making the penalization of incorrect transcriptions location-specific and context-dependent through policy gradients. Instead of uniform penalization, the policy gradient mechanism assigns different weights to different types of errors based on their impact on WER/CER metrics, allowing the model to learn nuanced error patterns where certain transcription errors are penalized more heavily than others depending on their proximity to the ground truth
Data Source
AI summary
The disclosed technology teaches a deep end-to-end speech recognition model, including using multi-objective learning criteria to train a deep end-to-end speech recognition model on training data comprising speech samples temporally labeled with ground truth transcriptions. The multi-objective learning criteria updates model parameters of the model over one thousand to millions of backpropagation iterations by combining, at each iteration, a maximum likelihood objective function that modifies the model parameters to maximize a probability of outputting a correct transcription and a policy gradient function that modifies the model parameters to maximize a positive reward defined based on a non-differentiable performance metric which penalizes incorrect transcriptions in accordance with their conformity to corresponding ground truth transcriptions; and upon convergence after a final backpropagation iteration, persisting the modified model parameters learned by using the multi-objective learning criteria with the model to be applied to further end-to-end speech recognition.