Selective Training for Ensemble Error Decorrelation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance machine learning ensembles tend to be overly correlated, leading to undesirable errors when multiple members make the same mistake on training data, which existing methods fail to effectively address.
Innovation Solution
A penalty term is added to the objective function of machine learning systems during training to penalize ensemble members for making the same mistakes, using a training node that connects to selected members and backpropagates penalties only when activated above a threshold, encouraging diversity in error correction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-performance classifiers are used in the ensemble, then individual classification accuracy is improved, but correlation between members increases leading to similar errors
Solution Approach 1:
The patent applies preliminary anti-action by introducing a penalty term into the training objective function that proactively prevents correlated errors before they occur. During training, when multiple ensemble members make the same error on a training example, a penalty is applied to discourage this correlated behavior, thereby improving ensemble robustness while maintaining individual classifier accuracy.
2Productivity
If standard training is applied to all ensemble members, then training efficiency is maintained, but correlated errors persist reducing ensemble performance
Solution Approach 1:
The patent applies local quality by selectively applying the penalty term only to specific training examples where correlated errors occur, rather than uniformly modifying all training processes. The penalty is locally applied to individual training examples based on whether multiple ensemble members make the same error, thereby maintaining overall training efficiency while improving error diversity where needed.
3Reliability
If penalty terms are added to penalize correlated errors, then error diversity is improved, but training complexity increases
Solution Approach 1:
The patent applies feedback by using the observed correlated errors during training to dynamically adjust the penalty term application. The system monitors which ensemble members make the same errors on which training examples and applies penalties accordingly, creating a feedback loop that guides the training process toward reduced error correlation without requiring complex manual intervention.
Data Source
AI summary
Computer-based systems and methods add extra terms to the objective function of machine learning systems (e.g., neural networks) in an ensemble for selected items of training data. This selective training is designed to penalize and decrease any tendency for two or more members of the ensemble to make the same mistake on any item of training data, which should result in improved performance of the ensemble in operation.