Neural Network Ensemble Training via Variance-Adjusted Dropout
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks often suffer from overfitting, especially as their complexity increases, leading to poor generalization to new examples due to memorization of training data, which existing ensemble learning methods fail to adequately address without significant engineering effort and computational resources.
Innovation Solution
The method employs variance-adjustable dropout training, which selectively disables hidden or input units in neural networks during training, creating an ensemble of networks with shared parameters, and introduces a hyper-parameter to adjust the variance of predictions, allowing for improved generalization and computational efficiency by preventing co-adaptations and regularizing the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the complexity of the neural network increases to model complicated relationships, then the accuracy on training data improves, but overfitting occurs and generalization to test data deteriorates
Solution Approach 1:
The patent merges multiple neural networks into an ensemble where networks share parameters through tying their weight matrices. This combination allows the system to maintain high training accuracy while improving generalization through the collective behavior of multiple networks, resolving the overfitting problem that plagues single complex networks
Solution Approach 2:
The patent introduces a regularization parameter λ that controls the strength of parameter sharing across networks in the ensemble. By adjusting this parameter, the system can balance between fitting training data and generalizing to test data, effectively managing the trade-off between training accuracy and generalization capability
2Reliability
If an ensemble of independent neural networks is used to improve accuracy and reduce overfitting, then generalization performance improves, but computational resources and engineering effort increase significantly
Solution Approach 1:
The patent combines multiple neural networks into a single computational structure by tying their parameters, allowing them to share computation. This merging reduces the computational burden compared to training and deploying fully independent networks, while still providing the benefits of ensemble learning for improved generalization
Solution Approach 2:
The shared parameters in the ensemble serve multiple networks simultaneously, making the parameters universal across the ensemble. This multi-functionality allows a single set of parameters to contribute to multiple networks' predictions, reducing the total computational resources required compared to independent networks
Data Source
AI summary
Systems and methods for training a neural network or an ensemble of neural networks are described. A hyper-parameter that controls the variance of the ensemble predictors is used to address overfitting. For larger values of the hyper-parameter, the predictions from the ensemble have more variance, so there is less overfitting. This technique can be applied to ensemble learning with various cost functions, structures and parameter sharing. A cost function is provided and a set of techniques for learning are described.