Neural Network Training via Hidden Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network training methods face challenges in efficiently training recurrent neural networks, particularly in preventing overfitting and co-adaptation of connection weights, which can lead to reduced recognition accuracy for unseen data.
Innovation Solution
The method involves selectively excluding a reference hidden node from the neural network during training, ignoring connections between this node and others in different time intervals, and randomly re-selecting the reference node for each item of sequential data, thereby training the network using a subset of hidden nodes to alleviate overfitting and prevent co-adaptation of weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all hidden nodes are used in training the recurrent neural network, then the network can capture complex temporal patterns, but overfitting and co-adaptation of connection weights occur leading to reduced generalization ability
Solution Approach 1:
The patent segments the hidden nodes into two groups: reference hidden nodes (excluded from training) and remaining hidden nodes (used in training). This segmentation allows the network to train on a subset of nodes while ignoring connections from reference nodes, thereby reducing overfitting and co-adaptation while maintaining the ability to capture temporal patterns through the remaining nodes.
Solution Approach 2:
The patent extracts reference hidden nodes from the complete set of hidden nodes and excludes them from the training process. By taking out these specific nodes and ignoring their connections to hidden nodes in different time intervals, the method reduces the complexity of the training problem and prevents overfitting, while the remaining nodes continue to learn effective temporal representations.
2Measurement precision
If connection weights are updated based on all hidden nodes, then the network learns comprehensive temporal dependencies, but co-adaptation of weights occurs reducing recognition accuracy for unseen data
Solution Approach 1:
The patent segments the connection weights into two categories: weights associated with remaining hidden nodes (updated during training) and weights associated with reference hidden nodes (ignored during training). This segmentation prevents co-adaptation by ensuring that reference node weights do not adjust in response to training data, thereby improving robustness to unseen data while maintaining recognition accuracy through proper weight updates on remaining nodes.
3Reliability
If the neural network uses full connectivity between hidden nodes across time intervals, then temporal pattern recognition is enhanced, but training efficiency decreases due to increased computational complexity
Solution Approach 1:
The patent extracts and removes connections from reference hidden nodes to hidden nodes in different time intervals from the training computation. By taking out these connections, the computational complexity of training is reduced while the remaining connected nodes continue to capture essential temporal patterns, thereby improving training efficiency without significantly compromising temporal pattern recognition capability.
Data Source
AI summary
A neural network training method based on training data, includes receiving training data including sequential data, and selecting a reference hidden node from hidden nodes in a neural network. The method further includes training the neural network based on remaining hidden nodes obtained by excluding the reference hidden node from the hidden nodes, and based on the training data, the remaining hidden nodes being connected with hidden nodes in a different time interval, and a connection between the reference hidden node and the hidden nodes in the different time interval being ignored.