Dynamic Model Acceptability Evaluation via Deviation Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI-based models for real-time data prediction lack the ability to dynamically evaluate the accuracy of predictions, leading to potential negative impacts on businesses due to decreased prediction accuracy caused by external factors.
Innovation Solution
A system that dynamically evaluates the acceptability of a model by generating predictions, calculating deviations between predicted and actual data, and adjusting downstream processes based on these deviations, allowing for partial acceptance of model predictions even when accuracy slips.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If AI-based models are used for real-time data prediction, then computing requirements and timing requirements are mitigated, but prediction accuracy decreases due to external factors
Solution Approach 1:
The system dynamically adjusts the acceptance of model predictions based on real-time deviation analysis. Instead of static acceptance/rejection, the system continuously monitors prediction accuracy against actual data and adapts its evaluation criteria, allowing flexible response to changing model performance conditions
Solution Approach 2:
The system implements a feedback mechanism where actual data points are compared with predicted values, deviations are calculated and analyzed, and this information feeds back into the model evaluation process. This closed-loop feedback enables continuous assessment and dynamic adjustment of prediction acceptance decisions
2Loss of energy
If model predictions are accepted when accuracy slips, then resource costs are reduced, but prediction reliability decreases
Solution Approach 1:
The system changes the parameter of prediction acceptance from a binary decision to a dynamic evaluation based on deviation curves. By transforming the acceptance criterion into a continuous function of deviation magnitude and trend, the system can adjust acceptance decisions based on multiple factors including the rate of deviation change, allowing partial acceptance even when accuracy slips occur
3Measurement precision
If dynamic evaluation of model acceptability is implemented, then prediction accuracy is maintained, but device complexity increases
Solution Approach 1:
The dynamic evaluation system is segmented into distinct functional modules: deviation calculation module, curve generation module, sampling module, and evaluation module. This segmentation allows each component to perform a specific function independently, making the overall complex system more manageable and maintainable while achieving accurate dynamic evaluation
Data Source
AI summary
Aspects of the invention include generating a plurality of predictions that each define a plurality of future inputs for a model. A deviation curve is generated by determining a distance between each prediction of the plurality of predictions and a respective known data point of a plurality of known data points. One or more points in the deviation curve are sampled and the sampled points are compared to a low threshold and a high threshold. A judgement is determined for each prediction to determine whether the respective prediction will be accepted or denied as an input to the model. The future inputs for the model are modified based on the judgments.