OSCTC Neural Network Training for Speech Recognition Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information classification systems, particularly in speech recognition, face challenges with high complexity, memory inefficiencies, and slow convergence rates, especially in resource-constrained devices like mobile phones, due to the use of traditional approaches that do not fully exploit the potential of recurrent neural networks (RNNs) and require unnecessary assumptions about data.
Innovation Solution
The introduction of One Spike Connectionist Temporal Classification (OSCTC) systems and methods for training recurrent neural networks using pre-segmented labeled data, which adaptively learn to improve convergence rates by determining momentum and computing root mean square gradients, and employ a combination of OSCTC and cross-entropy cost functions to optimize weight and bias updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional HMM-ANN combination approaches are used for speech recognition classification, then the system can handle sequential data structure, but the device complexity and computational requirements increase significantly
Solution Approach 1:
The patent extracts and removes the HMM component from the traditional HMM-ANN combination, using only the RNN to directly model sequential data. This eliminates the need for HMM assumptions and reduces system complexity while maintaining classification capability through the RNN's inherent sequential processing ability
Solution Approach 2:
The RNN is designed to perform multiple functions: it directly models the sequential structure of speech data, performs classification, and handles temporal dependencies all within a single unified architecture, replacing the need for separate HMM and ANN components
2Reliability
If conventional RNN training methods are used, then the network can learn from data, but the convergence rate is slow and training time is excessive
Solution Approach 1:
The patent applies preliminary actions by implementing adaptive learning rate mechanisms and momentum terms before actual training begins. The learning rate is dynamically adjusted based on gradient history, and momentum is computed in advance, allowing the network to converge faster from the start of training
Solution Approach 2:
The training process uses dynamic learning rates that adapt during training based on the computed momentum and gradient magnitude. The learning rate increases when gradients are small and decreases when gradients are large, creating a dynamic training process that accelerates convergence
3Reliability
If complex classification systems are implemented to improve accuracy, then classification performance improves, but memory efficiency and power consumption worsen
Solution Approach 1:
The patent employs simplified RNN architectures with basic activation functions and straightforward loss functions that require minimal computational resources. The model uses efficient operations that can be executed with low power consumption, making it suitable for mobile and embedded devices
4Productivity
If simplified models are used to reduce complexity, then computational efficiency improves, but flexibility and adaptability of the system decrease
Solution Approach 1:
The patent maintains model flexibility through adjustable parameters including the learning rate, momentum coefficient, and network architecture parameters. These parameters can be tuned to adapt to different speech recognition tasks and data characteristics while keeping the underlying model structure computationally efficient
Data Source
AI summary
A classification training system for binary and multi-class classification comprises a neural network operable to perform classification of input data, a training dataset including pre-segmented, labeled training samples, and a classification training module operable to train the neural network using the training dataset. The classification training module includes a forward pass processing module, and a backward pass processing module. The backward pass processing module is operable to determine whether a current frame is in a region of target (ROT), determine ROT information such as beginning and length of the ROT and update weights and biases using a cross-entropy cost function and One Spike Connectionist Temporal Classification (OSCTC) cost function. The backward pass module further computes a soft target value using ROT information and computes a signal output error using the soft target value and network output value.