Dimension-Level Anomaly Attribution in Streaming Data Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning and statistical algorithms designed for static data sets struggle to adapt and scale effectively for fast-growing and fast-changing streaming data, making it difficult to efficiently identify and explain anomalies, especially in high-dimensional data environments.
Innovation Solution
The implementation of a stream management and analytics service (SMAS) that utilizes an ensemble of random cut trees to generate anomaly scores and provide dimension-level attribution, allowing for real-time anomaly detection and explanation of why data points are considered anomalies through probabilistic 'fake' insert operations and bounding box-based computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional machine learning algorithms designed for static data sets are used, then algorithm simplicity is maintained, but adaptability to fast-growing and fast-changing streaming data deteriorates
Solution Approach 1:
The patent implements dynamic data structures (rolling histograms, sliding windows) that automatically adapt to changing streaming data characteristics. The anomaly detection system continuously updates its understanding of normal behavior patterns as new data arrives, making the algorithm inherently adaptive to fast-growing and fast-changing data without requiring complete retraining.
Solution Approach 2:
The patent segments the streaming data analysis into multiple independent components: rolling histograms for distribution tracking, sliding windows for recent pattern capture, and individual anomaly detection algorithms. This segmentation allows each component to be optimized independently while maintaining overall system adaptability to streaming data.
2Loss of information
If anomaly detection is performed quickly without explanations, then detection speed is improved, but information completeness deteriorates
Solution Approach 1:
The patent pre-computes and maintains rolling histograms and statistical summaries as data streams in. When an anomaly detection query arrives, the system can immediately compare against pre-computed baselines and provide explanations without performing heavy computations at query time, thus delivering both speed and completeness.
Solution Approach 2:
The patent introduces rolling histograms and statistical summaries as intermediary structures that capture the essence of normal data behavior. These intermediaries serve as reference points for anomaly detection, enabling the system to quickly explain anomalies by comparing against pre-computed distributions without recalculating from raw data.
3Measurement precision
If the number of data dimensions increases, then measurement precision is improved, but computational complexity increases
Solution Approach 1:
The patent computes rolling histograms and statistical summaries independently for each dimension rather than treating the multi-dimensional data as a single complex entity. This local quality approach allows the system to maintain high measurement precision across many dimensions while keeping computational complexity manageable by processing each dimension separately and efficiently.
Data Source
AI summary
At an artificial intelligence system, a random cut tree corresponding to a sample of a multi-dimensional data set is traversed to determine a tree-specific vector indicating respective contributions of individual dimensions to an anomaly score of a particular data point. Level-specific vectors of per-dimension contributions obtained using bounding-box analyses at each level during the traversal are aggregated to obtain the tree-specific vector. An overall anomaly score contribution for at least one dimension is obtained using respective tree-specific vectors generated from one or more random cut trees, and an indication of the overall anomaly score contribution is provided.