Sequential Ensemble Model Training for Open Set Diversity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional machine learning ensembles face challenges in achieving effective diversity, particularly in open-set problems, where the quality and accuracy of the ensemble degrade due to the trade-off between data fidelity and decorrelation components in existing loss functions.
Innovation Solution
The approach involves co-training or sequentially training machine learning models to focus on decorrelating negative results by encouraging inter-model disagreement on incorrect classes, using a combining function that merges probability vectors to enhance ensemble diversity without the trade-off, and employing feature engineering to ensure each model learns distinct features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional independent training with different training sets is used to achieve diversity, then model diversity is improved, but accuracy on open-set problems deteriorates due to the trade-off between data fidelity and decorrelation
Solution Approach 1:
The patent applies preliminary action by pre-processing training data into positive samples (from the training distribution) and negative samples (from out-of-distribution data) before training begins. This allows models to be explicitly exposed to both in-distribution and out-of-distribution examples during training, preparing them to handle open-set problems better while maintaining diversity through different sample assignments across ensemble members.
Solution Approach 2:
The patent implements local quality by assigning different roles to different models in the ensemble - some models focus more on positive samples while others focus more on negative samples. This is achieved through the loss function that combines data fidelity (for positive samples) and decorrelation (for negative samples) components with different weighting, allowing each model to develop specialized expertise while contributing to overall ensemble diversity.
2Reliability
If decorrelation components are added to loss functions to promote diversity, then ensemble diversity is improved, but the trade-off causes accuracy degradation on in-distribution data
Solution Approach 1:
The patent applies segmentation by dividing the training data into two distinct segments: positive samples (in-distribution) and negative samples (out-of-distribution). The loss function is also segmented, with the data fidelity component operating on positive samples and the decorrelation component operating on negative samples. This segmentation allows diversity to be promoted on negative samples without compromising accuracy on positive samples.
Solution Approach 2:
The patent extracts the decorrelation objective from the overall training process and applies it specifically to negative samples rather than applying it uniformly to all samples. By taking out the decorrelation component and restricting it to negative samples only, the patent eliminates the trade-off that would otherwise force a compromise between accuracy and diversity across all data.
3Reliability
If models are trained to disagree on incorrect classes, then reliability for open-set detection is improved, but complexity of the training process increases
Solution Approach 1:
The patent implements universality by designing a single loss function that simultaneously achieves multiple objectives: maintaining accuracy on in-distribution data through the data fidelity component, promoting diversity on out-of-distribution data through the decorrelation component, and enabling open-set detection. This multi-functional loss function approach consolidates what would otherwise require multiple separate training processes into one unified training procedure.
Data Source
AI summary
Disclosed are systems and method for training an ensemble of machine learning models with a focus on feature engineering. For example, the training of the models encourages each machine learning model of the ensemble to rely on a different set of input features from the training data samples used to train the machine learning models of the ensemble. However, instead of telling each model explicitly which features to learn, in accordance with the disclosed implementations, ML models of the ensemble may be trained sequentially, with each new model trained to disregard input features learned by previously trained ML models of the ensemble and learn based on other features included in the training data samples.