Predictive Model Anomaly Detection via Statistical Control Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a need for a systematic method to verify the accuracy of predictions made by predictive data analysis models and identify errors, such as incorrect input data, to ensure reliable outputs.
Innovation Solution
A method that feeds current and historical data sets into a predictive data analysis model to generate prediction results, determines class label distributions, and sets upper and lower control limits to alert for anomalies, allowing for model modification through user correction, retraining, or switching to alternative models to ensure predictions fall within defined limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If prediction models are used to generate predictions from input data, then productivity is improved, but reliability deteriorates due to lack of systematic verification
Solution Approach 1:
The patent implements a feedback mechanism by comparing current prediction results against historically established control limits. The system continuously monitors prediction outputs and triggers alerts when values fall outside acceptable ranges, enabling systematic verification of prediction reliability without compromising generation efficiency.
Solution Approach 2:
The system performs preliminary actions by pre-calculating control limits from historical data before actual prediction verification occurs. This preparation work includes determining upper and lower control limits based on historical class label distributions, enabling rapid reliability checking when predictions are generated.
2Reliability
If control limits are set to detect anomalies, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent applies parameter changes by establishing statistical control limits (upper and lower bounds) derived from historical data parameters. These parameter-based thresholds enable anomaly detection through simple comparison operations, avoiding the need for complex verification systems while maintaining high reliability.
3Measurement precision
If the model is modified through user correction or retraining, then prediction accuracy is improved, but loss of time increases
Solution Approach 1:
The system implements self-service by automatically detecting when predictions fall outside control limits and triggering alerts for model modification. This automated monitoring reduces the time loss associated with manual verification and enables timely model retraining only when necessary, rather than requiring continuous manual intervention.
Data Source
AI summary
A current data set with multiple records is fed into a data analysis model. The current data set is limited to data occurring in a current time window with a predetermined window size. The model is run on the current data set and a current data prediction result is generated. Limited historical data sets having multiple records are selected. Each record has values for several features. Each historical data set is limited to data occurring in a historical time window having the window size. A historical class label distribution is determined for the historical data sets and an upper and lower control limit are determined for the historical data sets using the historical class label distribution. A current class label distribution having a mean value is determined for the current prediction result. An alert is provided when the mean value is not between the upper and lower control limit.