Neural Network Training via Salience-Based Connection Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks tend to overfit training data, leading to incorrect classification of new data points, as they learn to classify instances from the training set more accurately than examples not in the training set, a problem known as overfitting.
Innovation Solution
The method involves training the artificial neural network by selecting a subset of connections based on salience values, which are determined by propagating training salience values through the network, and using these selected connections to determine activation values for subsequent layers, thereby reducing overfitting by emphasizing more relevant aspects of the training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the artificial neural network learns classification relationships from training data, then classification accuracy on training examples is improved, but classification accuracy on new unseen data deteriorates due to overfitting
Solution Approach 1:
The patent extracts and emphasizes only the most salient connections in the neural network based on salience values, removing less important connections from active participation in classification. This selective extraction prevents the network from learning spurious patterns while maintaining accuracy on important features, thereby reducing overfitting and improving generalization to unseen data.
Solution Approach 2:
The patent applies different treatment to different connections within the network based on their individual salience values. High-salience connections are emphasized and retained, while low-salience connections are suppressed or removed. This local differentiation allows the network to focus computational resources on the most important feature relationships, improving both training accuracy and generalization performance.
2Loss of information
If all connections in the neural network are used for classification, then comprehensive feature analysis is achieved, but computational complexity and processing time increase
Solution Approach 1:
The patent extracts only the most salient connections based on calculated salience values, removing redundant or less important connections from the computational process. This extraction maintains the essential information needed for accurate classification while significantly reducing the number of active connections, thereby lowering computational complexity and processing time without substantial loss of classification capability.
Solution Approach 2:
The patent applies partial action by activating only a subset of connections rather than all connections in the network. By using only the most salient connections (partial action), the system achieves efficient computation while maintaining sufficient information for accurate classification. This selective activation reduces computational burden while preserving the essential feature relationships needed for task performance.
Data Source
AI summary
A method of training an artificial neural network The artificial neural network comprising a plurality of connections connecting nodes arranged in at least an initial and a subsequent layer. The method comprises receiving a training example, the training example having input values, a target, and an associated training salience value indicating an importance of the input values to determining the target. The method further comprises determining salience values for nodes of the initial layer from the training salience value; determining an activation value for at least one node of the subsequent layer by propagating the input values using a subset of the connections selected based on the determined salience values for the nodes of the initial layer; and training the artificial neural network using the activation value, the trained artificial neural network configured to determine a relationship between the input data values and the target.