Multi-Period Forecast Scoring for Data Stream Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data monitoring systems face challenges in efficiently differentiating anomalous data values from non-anomalous ones in data streams, particularly when deviations occur due to server malfunctions or external events, leading to resource-intensive investigations.
Innovation Solution
A predictive modeling approach is employed, where a server computer selects models matching the time interval of new data values, generates scores based on historical data, and determines if the final score exceeds a threshold to identify anomalies, with the ability to create new models for adapting to changing patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If detailed investigation is performed to determine whether data deviation is due to server malfunction or external event, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system performs preliminary anomaly detection using statistical models and deviation analysis before detailed investigation. By pre-calculating expected values based on historical data and identifying deviations that exceed thresholds, the system filters out obvious anomalies and normal variations, reducing the need for detailed investigation of non-anomalous data points.
Solution Approach 2:
The patent replaces manual detailed investigation with automated computational analysis. Statistical models, deviation calculations, and threshold comparisons are performed automatically by the system, substituting human analysts' time-consuming manual examination with rapid automated processing that maintains detection accuracy while significantly reducing time loss.
2Measurement precision
If multiple statistical models are used to predict data values, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The system segments the prediction task into multiple independent statistical models, each handling specific aspects of data prediction. By dividing the complex prediction problem into separate models that can be independently developed, validated, and maintained, the system achieves high prediction accuracy while managing complexity through modular architecture.
Solution Approach 2:
The patent creates a universal framework that can accommodate multiple different statistical models (linear regression, exponential smoothing, ARIMA, etc.) through a common interface and evaluation mechanism. This multi-functional system allows various model types to be used interchangeably or in combination, achieving high prediction accuracy across different data patterns without proportionally increasing management complexity.
3Reliability
If anomaly detection is performed on all data deviations, then reliability is improved, but loss of energy increases
Solution Approach 1:
The system applies partial action by focusing computational resources only on data points that exhibit significant deviations from expected values. Rather than performing exhaustive analysis on all incoming data, the system calculates expected values and compares actual values against thresholds, applying full analytical power only when deviation exceeds the threshold, thus maintaining reliability while reducing energy consumption.
Solution Approach 2:
The patent dynamically adjusts detection parameters such as threshold values and sensitivity levels based on historical data characteristics and current system conditions. By changing these parameters adaptively, the system optimizes the balance between detection reliability and computational energy usage, reducing false positives and unnecessary deep analysis while maintaining high detection accuracy for genuine anomalies.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the invention are directed to techniques for detecting anomalous values in data streams using forecasting models. In some embodiments, a computer can receive a value of a data stream comprising a plurality of data values, where the received value corresponds to a time interval and previously received values each correspond to a previous time interval. Models can be selected based on the time interval, where each of the models has a different periodicity. For each of the selected models, the computer may generate a score by generating a prediction value based on the model and generating the score based on the prediction value and the received value. A final score can then be generated based on the scores. Next, a score threshold can be generated. If the final score exceeds the score threshold, the computer may generate a notification that indicates that the data value is an anomaly.