Neural Network Node Training with Amplified Gradients for Decisive Activations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face a trade-off between learning from training data and overfitting, which degrades performance on new data.
Innovation Solution
The system employs cooperative training of multiple generators and a classifier, using a learning coach to control hyperparameters and impose restrictions, such as regularization and smoothing estimates, to mitigate overfitting while aggressively learning from data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning systems aggressively learn from training data, then learning capacity is improved, but overfitting increases causing worse performance on new data
Solution Approach 1:
The patent segments the machine learning system into multiple independent generators, each trained to learn different aspects or features from the training data. This segmentation allows the system to aggressively learn from training data through multiple specialized components while the collective diversity of these segments prevents any single component from overfitting too severely, thus resolving the contradiction between learning capacity and generalization performance.
Solution Approach 2:
The patent introduces a learning coach as an intermediary component that mediates between the aggressive learning process and the need to prevent overfitting. The learning coach monitors the training process, controls the training dynamics, and ensures that the system learns effectively from training data while maintaining generalization capability to new data, thus resolving the contradiction.
2Loss of information
If multiple generators are trained aggressively from training data, then knowledge acquisition is improved, but overfitting increases
Solution Approach 1:
The patent merges multiple generators into a cooperative system where each generator is trained aggressively on different aspects of the training data. By combining their outputs and training them cooperatively with shared objectives, the system achieves comprehensive knowledge acquisition while the diversity and cooperation among generators prevent any single generator from severe overfitting, thus resolving the contradiction between knowledge acquisition and overfitting.
Solution Approach 2:
The patent implements feedback mechanisms where the learning coach monitors the training progress of multiple generators and adjusts their training dynamics accordingly. This feedback control ensures that generators continue to acquire knowledge aggressively while the learning coach intervenes to prevent overfitting by adjusting training parameters, thus resolving the contradiction between knowledge acquisition and overfitting.
Data Source
AI summary
Computer-implemented systems and methods improve training of a neural network. Whether a target node is not decisive on a training data item is determined. Upon a determination that the target node is not decisive, a partial derivative of an objective for the target node is multiplied by a factor greater than 1.0 for the training data item. Determining whether the target node is not decisive can comprise determining whether a direction of the derivative is in a direction that would cause an update of learned parameters for the network to increase the difference between the activation value of the first target node for the training data item and a neutral activation value for the target node.