Time-Series Anomaly Detection Using Inverted Index Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly and trend detection systems in time-series data are computationally intensive and dependent on pre-trained models, making them inefficient for real-time analysis across large-scale datasets, especially when dealing with unknown combinations of dimensions.
Innovation Solution
A distributed computer system that identifies anomalies or trends by selectively training forecasting models only on unique combinations of dimension labels identified as potential anomalies, allowing for real-time detection without pre-known dimensions and scaling to billions of data points with linear latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If forecasting models are trained on all possible combinations of dimension labels, then anomaly detection accuracy is improved, but computational cost and time consumption increase significantly
Solution Approach 1:
The patent segments the analysis process into two distinct phases: (1) a filtering phase that identifies candidate dimension combinations using aggregate value comparisons, and (2) a forecasting phase that applies computationally intensive models only to these candidates. This segmentation resolves the contradiction by separating the broad search space exploration from the precise anomaly detection, enabling real-time processing while maintaining accuracy where it matters most.
Solution Approach 2:
The patent applies partial action by performing complete forecasting analysis on only a subset of dimension combinations—specifically those identified as candidates through aggregate value comparisons. Instead of exhaustively analyzing all possible dimension combinations, the system performs excessive action (complete forecasting) only where necessary, thereby reducing overall computational cost while maintaining anomaly detection capability.
2Productivity
If pre-trained forecasting models are used for specific dimensions, then computational efficiency is improved, but adaptability to unknown dimension combinations deteriorates
Solution Approach 1:
The patent performs preliminary action by computing aggregate values for all dimension combinations before applying forecasting models. This pre-computation creates a filtered set of candidate combinations that can be quickly evaluated, enabling the system to adapt to any dimension combination requested while maintaining computational efficiency through the pre-established aggregate value structure.
Solution Approach 2:
The patent implements universality by designing a dimension-agnostic aggregate value computation framework that works for any combination of dimensions. The system computes aggregate values uniformly across all possible dimension combinations, then selectively applies forecasting to candidates, making the system adaptable to any dimension configuration without requiring pre-trained models for specific dimension sets.
3Reliability
If all dimension combinations are analyzed for anomalies, then detection completeness is improved, but system complexity and computation time worsen
Solution Approach 1:
The patent introduces aggregate value comparisons as an intermediary step between raw data and forecasting model application. This intermediary filtering mechanism identifies candidate dimension combinations based on aggregate value thresholds, reducing the input size for the complex forecasting stage while maintaining detection completeness for anomalies that meet the candidate criteria.
Solution Approach 2:
The patent segments the anomaly detection task into a candidate identification stage (using aggregate values) and a verification stage (using forecasting models). This segmentation reduces system complexity by dividing the monolithic analysis task into manageable phases, where each phase handles a specific subset of the work, thereby maintaining detection completeness without overwhelming system resources.
Data Source
AI summary
Implementations identify anomalous events from indexed events. An example system receives s dimension(s) for events, a test start time and a test duration defining a test interval. The system may identify a set of events matching the dimension(s). The set includes events occurring within a test interval or within one of at least two reference intervals. The system generates, for the test interval and the reference intervals, an aggregate value for each unique combination of dimension values in the set of events. The system selects at least one of the unique combination of dimension values for anomaly detection based on a comparison of the aggregate values for the reference intervals and the test interval, and performs anomaly detection on a historical time series for the selected unique combination of dimension values. The system may report any of the selected unique combination of dimension values identified as an anomaly.