Forecasting-Based Anomaly Detection for Periodic Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting anomalies in data streams from multiple heterogeneous sources is challenging due to the likelihood of errors, as significant deviations can be caused by either system failures or external events, making it difficult to determine whether anomalies warrant investigation without extensive analysis.
Innovation Solution
A monitoring service uses forecasting models to differentiate anomalous data values from non-anomalous ones by selecting models with matching periodicities, generating scores based on historical data, and determining if the new data value exceeds a score threshold, thereby identifying potential anomalies and triggering alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If detailed investigation is performed to determine whether deviations are due to system failures or external events, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system performs preliminary anomaly detection using forecasting models before detailed investigation is required. By pre-calculating anomaly scores based on historical data and pattern matching, the system prepares predictions that guide subsequent investigation efforts, reducing the time needed for detailed analysis while maintaining detection accuracy.
Solution Approach 2:
The patent replaces manual detailed investigation with automated forecasting-based anomaly detection. The system uses computational models to automatically identify and score potential anomalies, substituting the mechanical process of human investigation with an automated information processing system that provides rapid preliminary assessment.
2Measurement precision
If multiple forecasting models are used to differentiate anomalies from external events, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The system segments the anomaly detection process into distinct phases: data collection, model selection based on periodicity matching, score generation for each model, and final anomaly determination. By dividing the complex task into manageable segments with clear decision rules, the system handles multiple forecasting models without overwhelming complexity.
Solution Approach 2:
The system changes the parameter of model selection from arbitrary or manual choice to systematic selection based on periodicity matching. By using periodicity as the key parameter for model selection, the system automatically determines which forecasting models are appropriate for each data stream, reducing complexity while improving differentiation accuracy.
3Adaptability or versatility
If anomaly detection is performed on data streams from multiple heterogeneous sources, then adaptability is improved, but difficulty of detecting and measuring increases
Solution Approach 1:
The system implements a universal forecasting-based anomaly detection framework that can handle multiple heterogeneous data sources through a common methodology. The same periodicity-matching model selection and score generation process is applied across different data streams, regardless of their source or nature, providing adaptability while maintaining consistent detection procedures that reduce overall difficulty.
Data Source
AI summary
Embodiments of the invention are directed to techniques for detecting anomalous values in data streams using forecasting models. In some embodiments, a computer can receive a value of a data stream comprising a plurality of data values, where the received value corresponds to a time interval and previously received values each correspond to a previous time interval. Models can be selected based on the time interval, where each of the models has a different periodicity. For each of the selected models, the computer may generate a score by generating a prediction value based on the model and generating the score based on the prediction value and the received value. A final score can then be generated based on the scores. Next, a score threshold can be generated. If the final score exceeds the score threshold, the computer may generate a notification that indicates that the data value is an anomaly.