Forecasting Models for Data Stream Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Detecting anomalies in data streams from multiple heterogeneous sources is challenging due to the difficulty in distinguishing between errors and external events, requiring resource-intensive investigations to determine whether deviations are due to system failures or external factors.
Innovation Solution
Implementing a data monitoring system that uses forecasting models to differentiate anomalous values from non-anomalous ones by selecting models matching the time interval of new data values, generating scores based on historical data, and determining if the scores exceed a threshold, with the ability to create new models for captured patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If detailed investigation is performed to determine whether data deviations are due to system failures or external factors, then detection accuracy is improved, but time consumption and resource usage increase
Solution Approach 1:
The system automatically performs anomaly detection and classification by comparing new data values against historical patterns and multiple forecasting models, eliminating the need for manual detailed investigations. The system self-identifies whether deviations are anomalous or due to external events based on predictive modeling
Solution Approach 2:
The system pre-establishes multiple forecasting models with different periodicities before data arrives. When new data values are received, these pre-configured models can immediately evaluate them without requiring time-consuming setup or manual analysis, enabling rapid anomaly classification
2Measurement precision
If multiple forecasting models with different periodicities are used to evaluate data values, then detection accuracy is improved, but device complexity increases
Solution Approach 1:
The system divides the forecasting task into multiple independent models, each handling a specific periodicity (e.g., daily, weekly, monthly patterns). Each model is evaluated separately and contributes to the overall anomaly score, making the complex system manageable through modular segmentation
Solution Approach 2:
The system creates a universal framework that can handle multiple periodicities and data patterns using the same basic architecture. The forecasting models serve multiple purposes: they can detect anomalies, identify seasonal patterns, and adapt to changing data characteristics, reducing the need for separate specialized systems
Data Source
AI summary
Embodiments of the invention are directed to techniques for detecting anomalous values in data streams using forecasting models. In some embodiments, a computer can receive a value of a data stream comprising a plurality of data values, where the received value corresponds to a time interval and previously received values each correspond to a previous time interval. Models can be selected based on the time interval, where each of the models has a different periodicity. For each of the selected models, the computer may generate a score by generating a prediction value based on the model and generating the score based on the prediction value and the received value. A final score can then be generated based on the scores. Next, a score threshold can be generated. If the final score exceeds the score threshold, the computer may generate a notification that indicates that the data value is an anomaly.