Streaming Data Anomaly Detection via Windowed Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection models are not well-suited for real-time processing of streaming data and struggle to adapt to different anomaly types, as they were primarily designed for batch processing and cannot handle unbounded datasets with real-time latencies, making it difficult to detect anomalies effectively in large-scale business applications.
Innovation Solution
A system and method that utilizes a natural language API to construct data pipelines from streaming data, allowing for real-time anomaly detection by creating data windows, aggregating data points, and comparing them to reference aggregations using customizable and canned anomaly models, enabling flexible detection of various anomaly types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional anomaly detection models designed for batch processing are used, then manufacturing precision of anomaly detection is maintained, but productivity for real-time streaming data processing deteriorates
Solution Approach 1:
The patent segments the streaming data into fixed-time windows and processes each window independently through anomaly detection models. This segmentation enables real-time processing of streaming data while maintaining detection accuracy by analyzing manageable data portions with established batch-processing models.
Solution Approach 2:
The patent performs data aggregation and windowing operations before anomaly detection to prepare the streaming data in a format suitable for existing batch-processing models. This preliminary transformation allows traditional reliable models to process real-time data effectively.
2Measurement precision
If customized anomaly detection models are created for each anomaly type, then measurement precision for specific anomaly types is improved, but device complexity increases
Solution Approach 1:
The patent implements a unified anomaly detection framework that processes multiple anomaly types through a common architecture. The system uses standardized data windowing and aggregation mechanisms that work across different anomaly types, reducing the need for completely separate customized models while maintaining detection precision through type-specific configuration parameters.
3Productivity
If data is processed in real-time with streaming frameworks, then productivity is improved, but adaptability to different anomaly types deteriorates
Solution Approach 1:
The patent employs dynamic configuration parameters within the streaming processing framework that can be adjusted based on the specific anomaly type being detected. The system maintains real-time processing throughput while adapting detection thresholds, window sizes, and aggregation methods dynamically to match different anomaly characteristics.
Data Source
AI summary
A computer system receives an input corresponding to one or more data pipeline metrics and generates a first plurality of data windows from a first portion of a plurality of streaming data, the first plurality of data windows corresponding to a first data pipeline metric. The computer system generates a first plurality of data aggregations corresponding to the first plurality of data windows, wherein generating the first plurality of data aggregations includes aggregating a data associated with each of the first plurality of windows based on the first data pipeline metric. The computer system accesses a database to determine a first set of reference data aggregations that corresponds to a first data aggregation of the first plurality of data aggregations and determines whether the first data aggregation corresponds to an anomaly based on utilizing a model to compare the first set of reference data aggregations to the first data aggregation.