Neural Network Training Method Using Reference Hidden Node Exclusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network training methods face challenges in efficiently training recurrent neural networks for sequential data recognition, particularly in maintaining long-term memory values and preventing overfitting, which affects the network's ability to generalize and classify new input patterns effectively.
Innovation Solution
The method involves selecting a reference hidden node in each time interval, excluding it from the training process, and maintaining its long-term memory value across intervals, while connecting remaining hidden nodes across different time intervals, thereby updating connection weights based on the training results to enhance generalization and reduce overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all hidden nodes are trained using standard backpropagation, then the network learns patterns effectively, but the training time increases and overfitting occurs
Solution Approach 1:
The patent extracts and excludes a reference hidden node from the training process while maintaining its long-term memory value. This selective exclusion reduces the computational burden on the reference node, thereby reducing overall training time while preserving the network's recognition accuracy through the node's maintained memory function.
Solution Approach 2:
The patent segments the hidden layer into remaining hidden nodes and a reference hidden node with distinct functional roles. The reference node maintains long-term memory without standard training, while remaining nodes undergo normal training. This segmentation allows parallel processing paths that reduce training time while maintaining recognition accuracy.
2Adaptability or versatility
If standard training is applied to all hidden nodes, then the network adapts to training data, but overfitting occurs and generalization ability decreases
Solution Approach 1:
By extracting the reference hidden node from standard training and maintaining its value across time intervals, the patent prevents this node from overfitting to training data patterns. This extracted node serves as a stabilizing element that improves generalization ability while the remaining nodes continue to learn from training data.
Solution Approach 2:
The patent changes the training parameter for the reference hidden node by excluding it from standard backpropagation updates. This parameter change (from trained to maintained) creates a regularization effect that prevents overfitting and enhances the network's ability to generalize to new data.
3Reliability
If the reference hidden node is excluded from training, then overfitting is reduced, but the node's contribution to pattern recognition may be limited
Solution Approach 1:
The reference hidden node serves multiple functions: it maintains long-term memory values across time intervals, provides regularization to prevent overfitting, and still contributes to pattern recognition through its maintained values. This multi-functionality ensures that exclusion from training does not limit its recognition contribution.
Solution Approach 2:
The reference hidden node maintains continuous useful action by preserving its value across time intervals without being reset or updated through training. This continuity allows the node to contribute consistently to pattern recognition while preventing overfitting, balancing both generalization and accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is a neural network training method and apparatus, and recognition method and apparatus. The neural network training apparatus receives data and train a neural network based on remaining hidden nodes obtained by excluding a reference hidden node from hidden nodes included in the neural network, wherein the reference hidden node maintains a value in a previous time interval until a subsequent time interval.