Gradient Boosting Rashomon Models for Fairer, Consistent Predictions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current ensemble learning methods in gradient boosting lack effective mechanisms to explore Rashomon sets, which are crucial for understanding and mitigating predictive multiplicity and inconsistency among base models.
Innovation Solution
A method is introduced to generate competing models in Rashomon sets by iteratively training multiple machine learning models with residuals, selecting models based on desirable properties like fairness and interpretability, and deploying subsets for downstream tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple base models are combined in ensemble learning, then predictive accuracy is improved, but predictive multiplicity and inconsistency among models increase
Solution Approach 1:
The patent extracts and analyzes the Rashomon set of competing models that achieve similar predictive performance. By identifying and studying these competing models, the system can understand the sources of predictive multiplicity and inconsistency, then address them through targeted model selection and ensemble construction strategies.
Solution Approach 2:
The patent employs parameter changes by varying model parameters such as tree depth, number of leaves, and splitting criteria to generate diverse models within the Rashomon set. This allows exploration of multiple solutions with similar accuracy but different structures, enabling better understanding of predictive multiplicity.
2Reliability
If more base models are trained to reduce variance, then predictive consistency is improved, but computational complexity and training time increase
Solution Approach 1:
The patent applies partial action by training a limited number of models within the Rashomon set rather than exhaustively searching all possible models. By focusing computational resources on a representative sample of high-performing models, the system achieves sufficient predictive consistency without the full computational burden of exhaustive model generation.
Solution Approach 2:
The patent uses copying strategies by creating multiple models with similar structures and parameters that achieve comparable performance. These copied models vary in specific parameters to explore the Rashomon set, providing diverse predictions without requiring fundamentally different model architectures.
3Reliability
If models are selected based on accuracy alone, then predictive performance is optimized, but model interpretability and fairness are reduced
Solution Approach 1:
The patent applies local quality by evaluating models not just on overall accuracy but on specific local properties such as interpretability and fairness. Different aspects of model performance are weighted differently depending on the specific application requirements, allowing optimization of both predictive performance and interpretability simultaneously.
Solution Approach 2:
The patent implements feedback mechanisms that continuously evaluate model performance on accuracy, interpretability, and fairness metrics. Based on this feedback, the system iteratively refines model selection and ensemble construction, adjusting parameters and compositions to achieve balanced performance across multiple criteria.
Data Source
AI summary
A method may include: receiving a dataset comprising a plurality of samples and a loss function; training a first number first machine learning models using the dataset comprising, wherein each of the first machine learning models has a similar performance; selecting one of the first machine learning models with a smallest loss; computing a residual for each of the plurality of samples using the one first machine learning model; defining a new dataset comprising the plurality of samples and the residual for each samples; training the first machine learning model with the new dataset; generating a second plurality of machine learning models by repeating the selecting, the computing, the defining, and training for a number of boosting iterations; selecting a subset of the second plurality of machine learning model models having a specified property; and deploying the subset of second machine learning models to a downstream task.