Deep Neural Network Boosting for Stable Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks suffer from overfitting and lack of generalization due to their large capacity, and applying stochastic gradient boosting techniques has been computationally prohibitive and prone to training instability.
Innovation Solution
Implementing stochastic gradient boosting with a weighted loss function that disallows gradient backpropagation from correctly classified examples, using a scheduled boosting approach to prevent overfitting and focus on discriminative features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If stochastic gradient boosting is applied to deep neural networks, then model generalization is improved, but training stability deteriorates
Solution Approach 1:
The patent modifies the loss function by introducing a weighting mechanism that assigns zero weight to correctly classified examples. This parameter change in the loss computation resolves the contradiction by preventing overfitting to training data while maintaining training stability through controlled gradient updates.
Solution Approach 2:
The patent applies boosting selectively by disallowing gradient backpropagation only from correctly classified examples rather than all examples. This partial action approach improves generalization by focusing computational effort on misclassified samples while avoiding the instability that would result from boosting all samples equally.
2Productivity
If gradient backpropagation is allowed from all examples, then training speed is improved, but overfitting increases
Solution Approach 1:
The patent extracts and removes the gradient contribution from correctly classified examples by setting their weight to zero in the loss function. This takes out the harmful overfitting influence while preserving the training speed benefit by still processing all examples through the network, just without backpropagating gradients from correctly classified ones.
Solution Approach 2:
The patent applies different treatment to different subsets of training data by assigning zero weight locally to correctly classified examples while maintaining full weight for misclassified examples. This local quality differentiation enables faster training overall while preventing overfitting to the correctly classified portion of the data.
3Measurement precision
If deep neural networks are trained to maximize conditional probability, then predictive feature learning is improved, but learning of non-predictive statistics increases
Solution Approach 1:
Instead of maximizing conditional probability for all examples, the patent inverts the approach by minimizing the loss contribution from correctly classified examples. This inversion prevents the model from learning non-predictive statistics while maintaining focus on predictive features through the modified gradient updates.
Data Source
AI summary
Aspects described herein may allow for the application of stochastic gradient boosting techniques to the training of deep neural networks by disallowing gradient back propagation from examples that are correctly classified by the neural network model while still keeping correctly classified examples in the gradient averaging. Removing the gradient contribution from correctly classified examples may regularize the deep neural network and prevent the model from overfitting. Further aspects described herein may provide for scheduled boosting during the training of the deep neural network model conditioned on a mini-batch accuracy and/or a number of training iterations. The model training process may start un-boosted, using maximum likelihood objectives or another first loss function. Once a threshold mini-batch accuracy and/or number of iterations are reached, the model training process may begin using boosting by disallowing gradient back propagation from correctly classified examples while continue to average over all mini-batch examples.