Neural Network Activation Scaling for Overfitting Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks tend to overfit training data, leading to poor performance on unseen data and requiring extensive processing with large training datasets, and existing methods for addressing overfitting and sparsity are insufficient for effective learning.
Innovation Solution
The method involves determining and applying scaling factors based on the rank of activation values in the neural network nodes to update weights, incorporating a sparsity penalty value to inhibit excessive node responses and improve training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard training methods are used to improve classification accuracy on training data, then the neural network achieves high accuracy on training examples, but it overfits and performs poorly on unseen data
Solution Approach 1:
The patent modifies the training process by introducing scaling factors that change the activation values of nodes during training. These scaling factors are determined based on the rank of activation values, effectively changing the parameters of the neural network training to prevent overfitting while maintaining generalization capability
Solution Approach 2:
The patent applies different scaling factors to different nodes based on their individual activation value ranks. This local differentiation allows the training process to selectively adjust the contribution of individual nodes, preventing any single node from dominating and reducing overfitting
2Reliability
If large training datasets are used to improve generalization performance, then the neural network learns better relationships, but the processing time and computational resources increase significantly
Solution Approach 1:
By introducing scaling factors based on activation value ranks, the patent modifies the training dynamics to achieve better generalization with fewer training instances. This parameter change allows the network to learn more efficiently without requiring extensive training data and time
Solution Approach 2:
The patent replaces the traditional approach of using more data and longer training time with a mathematical transformation (scaling factors) that achieves the same goal of improving generalization more efficiently
3Reliability
If existing overfitting prevention methods are applied, then some regularization is achieved, but they are insufficient to effectively address both overfitting and sparsity issues
Solution Approach 1:
The patent introduces scaling factors as a new parameter that simultaneously addresses overfitting and sparsity. By ranking nodes based on activation values and applying appropriate scaling, the method effectively controls both the generalization capability and the sparsity of the neural network responses
Solution Approach 2:
The patent combines multiple concepts (activation value ranking, scaling factors, and sparsity control) into a composite training approach that achieves both overfitting prevention and sparsity management, which individual methods cannot accomplish alone
Data Source
AI summary
A system and method of training an artificial neural network. The method comprises determining an activation value for each node in a set of nodes of the artificial neural network, the activation values being determined by applying training data to the artificial neural network, and scaling the determined activation values for each of a plurality of the nodes in a portion of the artificial neural network. Each scaled activation value is determined using a scaling factor associated with a corresponding one of the plurality of nodes. Each scaling factor is determined based on a rank of the corresponding node. The method further comprises updating weights associated with each of the plurality of nodes in the portion of the artificial neural network using the determined scaled activation values to train the neural network.