Neural Network Training via Hidden Vector Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network models for speech recognition face challenges in accurately matching estimation hidden vectors with answer hidden vectors across time steps, affecting the training and inference processes, especially in recurrent neural networks (RNNs) used for sequential data processing.
Innovation Solution
The method involves determining weights based on differences between estimation and answer hidden vectors across time steps to adjust model parameters, using a beam search to rank candidates, and applying these weights to reduce loss, ensuring that estimation hidden vectors match answer hidden vectors and estimation output vectors match answer output vectors, thereby improving the training and inference accuracy of RNNs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional training methods are used to train neural network models, then the model can learn mappings between input and output patterns, but the estimation hidden vectors do not accurately match the answer hidden vectors across time steps
Solution Approach 1:
The patent implements a feedback mechanism where the difference between estimation hidden vectors and answer hidden vectors is calculated and used to generate weighted loss signals. These feedback signals are then propagated back through the RNN to adjust weights and biases, continuously improving the matching accuracy of hidden vectors across time steps
Solution Approach 2:
The patent introduces dynamic parameter adjustments by applying different weights to loss calculations based on the magnitude of differences between estimation and answer hidden vectors. When differences are large, higher weights are applied to prioritize correction, and when differences are small, lower weights are used, effectively changing training parameters adaptively
2Measurement precision
If beam search is used to rank candidates during inference, then the most suitable candidates can be identified, but the computational complexity and search time increase
Solution Approach 1:
The patent applies partial action by limiting the beam search to a predetermined number of top candidates rather than evaluating all possible candidates. This partial evaluation approach maintains sufficient ranking accuracy for speech recognition while significantly reducing the computational burden and search time compared to exhaustive search
Data Source
AI summary
A method and apparatus for training a recognition model and a recognition method and apparatus using the model are disclosed. The apparatus for training the model obtains an estimation hidden vector output from a hidden layer of the model in response to an estimation output vector output from the model at a previous time being input into the model at a current time, and trains the model such that the estimation hidden vector of the current time matches an answer hidden vector output from the hidden layer in response to an answer output vector, corresponding to the estimation output vector of the previous time, being input into the model at the current time.