Data Reweighting for Machine Learning Model Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models struggle to accurately predict future outcomes when future conditions differ significantly from historical data, leading to a gap between expected and actual performance due to evolving trends or disruptive events.
Innovation Solution
A system and method for reweighting historical data by binning observation records based on predictive features and dependent variables, generating a target distribution, and optimizing weights to simulate future-like conditions, using an optimization engine to create a reweighted data sample that aligns with anticipated future scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional model assessment methods using historical data are used, then model assessment is simple and straightforward, but model performance prediction becomes inaccurate when future conditions differ from historical data
Solution Approach 1:
The patent applies preliminary action by pre-processing the historical assessment data before model evaluation. Specifically, it calculates reweighting factors based on the distribution differences between historical training data and future deployment data, then applies these weights to adjust the assessment results. This preliminary adjustment ensures that the model performance assessment reflects future conditions more accurately without requiring actual future data.
Solution Approach 2:
The patent changes the parameter of data weighting in the assessment process. It introduces reweighting factors that modify the importance of different historical samples based on how well their feature distributions match the expected future distribution. Samples with distributions closer to future conditions receive higher weights, while those with larger deviations receive lower weights, thereby improving prediction accuracy.
2Adaptability or versatility
If historical data is used directly for model training and assessment, then data usage is simple, but the model fails to adapt to evolving trends and disruptive events
Solution Approach 1:
The patent changes the distribution parameters of historical data by applying reweighting factors. These factors are calculated based on the divergence between historical data distribution and expected future distribution across various features. By adjusting the weights of individual samples or feature combinations, the model learns from historical data while adapting to anticipated future conditions, thereby improving versatility without requiring retraining on actual future data.
3Measurement precision
If data reweighting based on target distribution is applied, then model performance prediction accuracy improves, but computational complexity increases
Solution Approach 1:
The patent segments the data reweighting process into distinct steps: first calculating reweighting factors based on overall distribution differences, then optionally applying separate reweighting for different features or sample groups. This segmentation allows the computational workload to be distributed and managed more efficiently, while still achieving accurate future performance assessment through the cumulative effect of multiple targeted adjustments.
Data Source
AI summary
A computer-implemented method includes maintaining a historical data sample comprising a plurality of observation records, each observation record comprising a set of predictive features, a set of dependent variables, and a baseline weight variable; generating a target distribution for the plurality of observation records, wherein the target distribution comprises a first plurality of target percentages for a subset of the predictive features and a second plurality of target percentages for a subset of the dependent variables, generating a reweight variable for each observation record based at least in part on the target distribution and the baseline weight variable, wherein the reweight variable comprises optimized weights for each of the predictive features and each of the dependent variables, and generating a reweighted data sample by replacing the baseline weight variable in each observation record of the historical data sample with a corresponding reweight variable.