AI Model Validation via Short-Term Prediction Error
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial intelligence models face inaccuracies due to differences between the data used for prediction and learning, leading to mistakes, such as predicting holiday traffic based on weekend data.
Innovation Solution
A system that runs both a long-term and a short-term AI model on the same data set, with the short-term model providing a validation of the long-term model's predictions by calculating probabilities and errors, allowing for the selection of a representative AI model based on these calculations to improve prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Duration of action of moving object
If a long-term AI model is used for prediction, then the prediction coverage period is extended, but the prediction accuracy decreases due to data distribution differences
Solution Approach 1:
A short-term AI model is introduced as an intermediary validation mechanism. The short-term model runs on recent data to validate whether the long-term model's predictions are reliable for current conditions. This intermediary model bridges the gap between long-term predictions and current data distributions, allowing the system to extend prediction coverage while maintaining accuracy through validation.
2Adaptability or versatility
If data from different distributions (e.g., weekend data) is used to predict different scenarios (e.g., holiday traffic), then the prediction scope is expanded, but the prediction accuracy deteriorates
Solution Approach 1:
The system implements feedback by using the short-term model's predictions and accuracy metrics to validate the long-term model's predictions. The short-term model provides feedback on whether current data distributions align with training data, allowing the system to detect when predictions may be unreliable due to distribution shifts between different scenarios.
3Reliability
If multiple AI models are run to validate predictions, then the prediction reliability improves, but the computational complexity increases
Solution Approach 1:
Instead of running multiple long-term models for validation, the system uses a partial action approach by employing a shorter-duration model for validation purposes. This excessive action of running an additional model is justified because the short-term model requires fewer computational resources than multiple long-term models would, achieving reliability improvement with manageable complexity.
Data Source
AI summary
A system, method, and computer program product are provided for validating artificial intelligence models. In operation, a data set is obtained. At least one first artificial intelligence (AI) model on the data set is run, for a first time period, to output first predictions. Additionally, at least one second AI model on the data set is run, for a second time period that is less than the first time period, to output second predictions. A first probability of the first predictions and an error of the second predictions are calculated. Further, a second probability of the first predictions is calculated based on the first probability and the error. A representative AI model of the at least one first AI model is selected based on the second probability.