Statistical Closed-Form Isolation Forest for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods in computing systems are computationally intensive and unstable, particularly when using isolation forest algorithms, which require training models and traversing tree structures to identify anomalies, making them inefficient for rapid detection and remediation.
Innovation Solution
The implementation of a statistical closed-form isolation forest analysis that predicts the number of operations needed to isolate a data point from a historical time-series data set, allowing for deterministic and rapid identification of anomalous activity without the need for model training or tree traversal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If isolation forest algorithms are used for anomaly detection, then anomaly detection capability is improved, but processing time and computational resources increase
Solution Approach 1:
The patent uses pre-computed statistical summaries (mean, standard deviation, min, max) from historical data as lightweight, disposable proxies for the complex isolation forest model. These statistical objects are cheap to compute and store, replacing the need for expensive model training and traversal during actual anomaly detection, thus reducing processing time while maintaining detection capability
Solution Approach 2:
The patent transforms the anomaly detection problem from tree-based structural analysis to statistical parameter comparison. By changing the detection parameters from tree traversal depth and path length to simple statistical deviations (z-scores, interquartile ranges), the system achieves faster detection with reduced computational overhead while preserving anomaly detection effectiveness
2Reliability
If isolation forest algorithms are used for anomaly detection, then anomaly detection capability is improved, but computational complexity increases
Solution Approach 1:
The patent replaces the mechanical tree traversal process of isolation forests with statistical calculations. Instead of recursively partitioning data through tree structures, the system uses direct statistical comparisons (mean, std dev, percentiles) which are computationally simpler and avoid the complexity of model training, tree building, and traversal
Solution Approach 2:
The patent creates simplified copies of the anomaly detection function using statistical summaries of historical data. These statistical copies (pre-computed mean, variance, min, max values) replicate the anomaly detection capability of isolation forests without requiring the full computational apparatus of the original algorithm, reducing device complexity
3Measurement precision
If model training and tree traversal are performed, then anomaly detection accuracy is improved, but processing speed decreases
Solution Approach 1:
The patent performs preliminary computation of statistical summaries (mean, standard deviation, min, max) from historical data during offline processing. This preliminary action captures the essential patterns needed for anomaly detection without requiring real-time model training or tree traversal, enabling fast online detection while preserving accuracy
Solution Approach 2:
The patent extracts only the essential statistical features (mean, variance, min, max) from historical data that are necessary for anomaly detection, discarding the complex tree structures and training processes. This extraction maintains detection accuracy by preserving key data characteristics while eliminating computational overhead
Data Source
AI summary
Certain embodiments of the present disclosure provide techniques for detecting anomalous activity in a computing system. The method generally includes receiving a request to perform an action in a computing system. The request is added to a historical time-series data set. A portion of the historical time-series data set is selected for use in determining whether the received request is an anomalous request, and a set of previously identified outliers are removed from the selected portion of the historical time-series data set. An anomaly score is calculated based on a statistical analysis of the received request and the selected portion of the historical time-series data set, wherein the anomaly score comprises a predicted number of operations executed to isolate the received request from the selected portion of the historical time-series data set. One or more actions are taken to process the received request based on the calculated anomaly score.