Recurrent Neural Network Training Using Region of Target Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information classification systems, particularly in speech recognition, face challenges with high complexity, memory inefficiencies, and slow convergence rates, especially in resource-constrained devices like mobile phones, due to the use of traditional approaches that do not fully exploit the potential of recurrent neural networks (RNNs) and require unnecessary assumptions about data.
Innovation Solution
The proposed method involves training a recurrent neural network using pre-segmented labeled data with a Connectionist Temporal Classification (CTC) algorithm, employing a Region of Target (ROT) to focus updates and adaptively learning to improve convergence rates, utilizing a cross-entropy cost function and momentum for weight updates, and computing root mean square gradients to optimize network performance for binary and multi-class classification tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional HMM-ANN approaches are used for speech recognition classification, then the system can handle sequential data with long-range dependencies, but the device complexity and computational resources required increase significantly
Solution Approach 1:
The patent extracts and removes the HMM component from the traditional HMM-ANN architecture, using only the RNN to directly model sequential dependencies. This eliminates the need for HMM assumptions and reduces system complexity while maintaining the ability to handle long-range sequential relationships through the RNN's inherent temporal modeling capabilities
Solution Approach 2:
The RNN is designed to perform multiple functions: it directly models sequential dependencies, performs classification, and handles variable-length sequences without requiring separate HMM modules. This multi-functional approach reduces overall system complexity while maintaining classification reliability
2Reliability
If traditional RNN-HMM combined approaches are used, then the system can model sequential data with temporal structure, but the training complexity and computational overhead increase
Solution Approach 1:
The patent removes the HMM component from the training pipeline, using only the RNN for both training and inference. This simplifies the training process by eliminating the need to train separate HMM parameters and the complex interactions between HMM and RNN, while the RNN alone maintains sequential data modeling capability through its recurrent architecture
Solution Approach 2:
The RNN is pre-trained directly on the sequential data with classification targets, establishing temporal relationships and classification boundaries in a single unified training process. This preliminary direct training eliminates the need for subsequent HMM parameter estimation and alignment procedures, reducing overall training complexity
3Reliability
If full RNN training is applied without selective region updates, then the model can learn from all data uniformly, but the convergence rate decreases and training time increases
Solution Approach 1:
The patent segments the sequence data into distinct regions: ROT regions containing target information and non-ROT regions. This segmentation allows selective application of different update strategies, focusing computational effort on ROT regions to improve convergence while maintaining model learning completeness through comprehensive coverage of all regions
Solution Approach 2:
The patent applies different update qualities to different regions: full gradient updates are applied to ROT regions where target information exists, while simplified or skipped updates are applied to non-ROT regions. This local differentiation improves convergence rate by concentrating computational resources on informative regions while maintaining overall model learning through selective updates
4Reliability
If comprehensive weight updates are performed across all network parameters, then the model achieves better performance, but the memory usage and computational overhead increase
Solution Approach 1:
The patent segments the parameter update process by region, storing and updating only the necessary weight parameters associated with ROT regions in detail, while using simplified representations for non-ROT regions. This segmentation reduces memory usage by avoiding redundant storage of full gradient information across all regions while maintaining model performance through focused updates on critical regions
Solution Approach 2:
The patent extracts and removes unnecessary weight update computations for non-ROT regions, updating only the essential parameters that contribute to classification performance. This extraction reduces memory overhead by eliminating storage and computation of redundant gradient information while preserving model performance through targeted parameter updates
Data Source
AI summary
A classification training system for binary and multi-class classification comprises a neural network operable to perform classification of input data, a training dataset including pre-segmented, labeled training samples, and a classification training module operable to train the neural network using the training dataset. The classification training module includes a forward pass processing module, and a backward pass processing module. The backward pass processing module is operable to determine whether a current frame is in a region of target (ROT), determine ROT information such as beginning and length of the ROT and update weights and biases using a cross-entropy cost function and connectionist temporal classification cost function. The backward pass module further computes a soft target value using ROT information and computes a signal output error using the soft target value and network output value.