Forecasting Model Selection via Accuracy and Robustness Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing forecasting models face challenges in accurately selecting predictions when encountering anomalous predictor variable data, as conventional methods assume a correlation between model accuracy and robustness, which may not always hold, leading to inverse correlations and dampened impact of anomalous data on overall accuracy scores.
Innovation Solution
A system and method that evaluates and selects predictions based on both accuracy and robustness scores, distinguishing between normal and anomalous time periods to choose models that excel in respective domains, ensuring accurate forecasting in both conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional forecasting methods assume correlation between model accuracy and robustness, then model selection is simplified, but prediction accuracy deteriorates when anomalous data is present
Solution Approach 1:
The patent segments the forecasting evaluation into two distinct components: accuracy scores for normal time periods and robustness scores for anomalous time periods. This segmentation allows the system to independently evaluate model performance in different data conditions, preventing the inverse correlation between accuracy and robustness from degrading overall prediction accuracy.
Solution Approach 2:
The patent introduces a new parameter dimension by evaluating both accuracy and robustness scores separately, then combining them through weighted aggregation. This parameter change enables the system to capture the inverse correlation between accuracy and robustness while maintaining reliable predictions through balanced score integration.
2Measurement precision
If model accuracy is prioritized over robustness, then predictions improve in normal conditions, but performance deteriorates in anomalous conditions
Solution Approach 1:
The patent segments time periods into normal and anomalous categories, evaluating model performance separately for each segment. This allows high-accuracy models to be selected for normal periods while robust models are selected for anomalous periods, resolving the trade-off between accuracy and reliability across different conditions.
Solution Approach 2:
The patent applies local quality by assigning different evaluation criteria to different time period segments. Normal time periods use accuracy scores as the primary criterion, while anomalous time periods use robustness scores, ensuring each segment is evaluated by the most appropriate metric for its characteristics.
3Measurement precision
If anomalous data impact on overall accuracy scores is dampened, then overall accuracy metric becomes less sensitive to model robustness, but model selection fails to account for anomalous conditions
Solution Approach 1:
The patent segments the overall evaluation into separate accuracy and robustness score calculations, preventing anomalous data from being dampened in the overall accuracy metric. This ensures model selection remains sensitive to robustness while maintaining accurate measurement of normal period performance.
Solution Approach 2:
The patent introduces weighted score aggregation as an intermediary mechanism that combines accuracy and robustness scores. This intermediary allows the system to maintain sensitivity to both normal and anomalous conditions while producing a unified model selection criterion that accounts for model adaptability across different data conditions.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for selecting predictions by models. A computer receives a request for a forecast of a dependent variable in a time domain, where the time domain includes first time periods that have normal labels due to normal predictor variable data and second time periods that have anomalous labels due to anomalous predictor variable data. The computer retrieves accuracy scores and robustness scores of models, where the accuracy scores indicate forecasting accuracy in the first time periods and the robustness scores indicate forecasting accuracy in the second time periods. For predictions in the first time period, the computer selects dependent variable values predicted by a first model that has highest values of the accuracy scores. For predictions in the second time periods, the computer selects dependent variable values predicted by a second model that has highest values of the robustness scores.