Multi-Phase ML Training with Weighted Data for Upper Distribution Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning model training techniques face challenges in accurately classifying inputs at the upper end of the probability distribution, particularly in binary classification problems, due to global optimization methods that prioritize consistency across all prediction values, leading to skewed training datasets and reduced model efficacy.
Innovation Solution
A multi-phase training technique is employed, where a first phase trains a classification model with equal weighting of training samples, followed by a second phase that weights samples based on their model scores to emphasize the upper end of the probability distribution, refining the model's parameters and improving accuracy in the relevant classification range.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If global optimization training is used to maintain consistency across all prediction values, then model accuracy is relatively consistent across the entire distribution, but the model's ability to accurately classify new observations at the upper end of the probability distribution is limited
Solution Approach 1:
The training process is divided into two distinct phases: Phase 1 trains the model on the full training dataset to achieve general consistency, while Phase 2 focuses specifically on the upper end of the probability distribution by selecting and re-weighting samples. This segmentation allows the model to first learn general patterns and then specialize in high-stakes classification regions.
Solution Approach 2:
The patent applies different training strategies to different regions of the probability distribution. In Phase 2, samples are re-weighted based on their predicted probabilities, with higher weights assigned to samples in the upper end of the distribution. This local quality adjustment ensures that the model pays special attention to the classification region where accurate decision-making is most critical.
2Reliability
If training data is weighted to emphasize the upper end of the probability distribution, then classification accuracy at the upper end improves, but accuracy at the lower end may decrease
Solution Approach 1:
Phase 1 serves as a preliminary training stage where the model learns general classification patterns from the full dataset with uniform weighting. This preliminary action ensures that the model has a solid foundation before the specialized Phase 2 training begins, preventing complete loss of lower-end accuracy while focusing on improving upper-end performance.
Solution Approach 2:
Phase 2 applies partial action by focusing training efforts only on the upper end of the probability distribution rather than the entire range. The re-weighting scheme selectively emphasizes samples with higher predicted probabilities, accepting that some degradation in lower-end accuracy may occur but is tolerable since those regions are less critical for final classification decisions.
3Productivity
If equal weighting is applied to all training samples, then training is simpler and faster, but the model fails to accurately classify inputs at the upper end of the probability distribution
Solution Approach 1:
The training process is segmented into two phases with different weighting strategies. Phase 1 maintains equal weighting for simplicity and speed, while Phase 2 introduces selective weighting to improve upper-end accuracy. This segmentation allows the system to benefit from both simple/fast training and high accuracy in critical regions.
Solution Approach 2:
The patent changes the weighting parameter dynamically between phases. In Phase 1, all samples have equal weight (w=1). In Phase 2, weights are modified based on predicted probabilities (w = p^α where p is predicted probability and α is a hyperparameter). This parameter change allows the model to adapt to different training objectives without completely retraining from scratch.
Data Source
AI summary
Techniques are disclosed relating to multi-phase training of machine learning models using weighted training data. In some embodiments, a computer system may train a machine learning classification model in at least two phases. During an initial training phase, the computer system may train an initial version of the classification model based on a training dataset, applying equal weight to the training samples in the training dataset. The computer system may then generate model scores for the training samples using the initial version of the classification model. Based on these model scores, the computer system may generate, for the training samples, corresponding weighting values. The computer system may then perform a subsequent training phase to generate an updated version of the classification model, where, during this subsequent training phase, at least some of the training samples are weighted using their respective weighting values.