Streaming Histogram Anomaly Detection Under Fixed Resource Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream processing technologies face challenges in performing anomaly detection with low latency and resource efficiency, particularly when dealing with large windows of events, as they require reprocessing large datasets and consume excessive resources, failing to meet real-time requirements.
Innovation Solution
A device and method utilizing a specialized stream compute unit with incremental histogram updates and a flexible learning mode, employing a streaming random forest approach that maintains a fixed resource footprint and supports supervised, semi-supervised, and unsupervised learning, enabling efficient anomaly detection with low latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional stream processing reprocesses large windows of events to perform anomaly detection, then detection accuracy is improved, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent pre-computes and stores aggregate statistics (count, sum, mean, variance) for each histogram bin as events arrive, rather than reprocessing raw events during anomaly detection. This preliminary aggregation enables fast anomaly scoring by comparing new events against pre-computed baseline statistics, resolving the contradiction between detection accuracy and processing speed
Solution Approach 2:
The patent creates a compressed statistical representation (copy) of the event window data in the form of histograms with aggregate features. Instead of storing and reprocessing all raw events, it maintains a compact copy containing essential statistical properties, enabling efficient anomaly detection without sacrificing detection accuracy
2Measurement precision
If traditional stream processing holds all events in memory for window operations, then complete data analysis is achieved, but memory consumption increases exponentially with window size
Solution Approach 1:
The patent extracts essential statistical features (count, sum, mean, variance) from the raw event data and stores only these aggregated values in histogram bins. This extraction process eliminates the need to hold all raw events in memory, reducing memory consumption from O(N) to O(K) where K is the number of histogram bins, while preserving detection completeness through sufficient statistical representation
Solution Approach 2:
The patent transforms the raw event data into a different parameter space by computing aggregate statistics (count, sum, mean, variance) for each bin. This parameter transformation compresses the data representation from individual event records to summary statistics, dramatically reducing memory requirements while maintaining the ability to detect anomalies through statistical comparison
3Loss of time
If stream processing re-computes window functions at each triggering moment, then real-time detection is achieved, but computation time becomes prohibitive for large windows
Solution Approach 1:
The patent continuously updates histogram counts and aggregate statistics incrementally as events arrive, rather than re-computing from scratch at each triggering moment. This continuous incremental updating maintains current anomaly detection capability with minimal computation per event, achieving real-time detection with constant rather than linear computation power requirements
Solution Approach 2:
The patent pre-computes aggregate statistics (sum, mean, variance) and stores them with the histogram counts. This preliminary computation eliminates the need for expensive re-computation at triggering moments, as the anomaly detection function can directly use pre-computed statistics to quickly assess whether new events represent anomalies, dramatically reducing computation power requirements while maintaining real-time detection
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to the field of artificial intelligence, machine learning and Big Data, and provides a device (100) and method (300) for anomaly detection, performed in a stream of events. To this end, the device 100 for anomaly detection on an input stream (101) of events (102) is provided, wherein each event (102) comprises a value (103). The device (100) comprises a modeling unit (104), configured to represent each event 102 from the input stream (101) of events (102) into at least one histogram (105) over a predefined tree data structure (106), to obtain a model 107 of the input stream (101); a change detection and adaptation module (108), configured to detect a long term change (109) of the distribution of the events (102) in the input stream 101 based on the model (107) of the input stream (101); and a scoring unit (110), configured to calculate an anomaly score (111), representing the probability of an anomaly in the input stream (101) of events (102), based on the model (107) of the input stream (101), and the detected long term change (109).