Prediction Accuracy Model for Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine-learning models experience a drop in accuracy over time, and re-training with new data is challenging due to the costs associated with generating and processing new training data.
Innovation Solution
A system that generates prediction training data based on existing model predictions and probabilities, using a prediction accuracy model to determine the accuracy of these predictions and automatically correct them, thereby improving model accuracy without the need for a proxy model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If re-training the machine-learning model with new training data, then model accuracy is improved, but the cost and complexity of generating and processing new training data increases
Solution Approach 1:
The patent introduces a prediction accuracy model as an intermediary component that assesses the quality of predictions made by the deployed machine-learning model. This intermediary model evaluates prediction confidence and identifies uncertain cases, enabling selective re-training only on necessary data points rather than requiring comprehensive new training datasets. The prediction accuracy model acts as a mediator between the deployed model and the re-training process, reducing the complexity of data generation and processing.
Solution Approach 2:
The patent changes the parameter being monitored from raw prediction outputs to prediction accuracy metrics. By training a separate prediction accuracy model that evaluates the confidence and correctness of predictions, the system transforms the approach to model maintenance. Instead of continuously generating new training data, the system uses the prediction accuracy model to identify when and what to re-train, changing the parameter focus from data volume to data quality assessment.
2Reliability
If generating new training data and re-training machine-learning models, then model accuracy is improved, but the time and resources required increase
Solution Approach 1:
The patent implements preliminary action by training a prediction accuracy model in advance that can quickly assess prediction quality. This pre-trained accuracy model enables rapid evaluation of new predictions without requiring immediate re-training of the main model. The system performs preliminary assessment of prediction confidence before committing resources to full re-training, saving time by filtering out cases that don't require intervention.
Solution Approach 2:
The patent applies partial action by re-training the machine-learning model only on specific subsets of data identified as problematic by the prediction accuracy model, rather than re-training on all available data. This selective approach reduces the time and computational resources required for re-training while still achieving accuracy improvements where needed.
3Reliability
If using a proxy model to identify new training data, then model accuracy can be maintained, but the system complexity and computational cost increase
Solution Approach 1:
The patent extracts the accuracy assessment function from the main machine-learning model by implementing a separate prediction accuracy model. This extracted component specifically handles the evaluation of prediction quality and identification of training needs, allowing the main model to focus on its primary function. The separation reduces computational overhead by dedicating specific functionality to specialized components.
Solution Approach 2:
The prediction accuracy model serves multiple functions: it assesses prediction confidence, identifies uncertain cases, determines re-training needs, and provides quality metrics. This multi-functional component replaces the need for multiple separate systems that would otherwise be required to perform these individual tasks, reducing overall computational cost while maintaining accuracy.
Data Source
AI summary
Embodiments are disclosed for a method for machine-learning model accuracy. The method includes generating prediction training data based on training predictions and corresponding probabilities of the training predictions. A classifier of a machine-learning model generates the training predictions. The method also includes training a prediction accuracy model to determine whether the training predictions generated by the machine-learning model are correct. Additionally, the method includes generating predictions in response to corresponding client transactions for the machine-learning model. Further, the method includes determining whether the predictions are accurate using the prediction accuracy model. Also, the method includes providing client predictions corresponding to the client transactions based on the determination.