Anomaly Detection via Multivariable KL Divergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection techniques in computer systems suffer from high false-positive rates, inefficiency in handling high-dimensional data, and inability to detect anomalies based on correlation patterns between variables, leading to missed anomalies and unnecessary resource waste.
Innovation Solution
A multivariable Kullback-Leibler (KL) divergence-based approach that compares current system observations to historical data, accounts for missing samples, and detects anomalies across multiple variables in a single iteration, enabling correlation-based anomaly detection and dynamic range adjustments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional anomaly detection techniques are used, then detection speed is maintained, but false-positive rate increases and reliability decreases
Solution Approach 1:
The patent transforms anomaly detection from point-wise comparisons to probability distribution comparisons using KL divergence. By changing the parameter space from individual data points to probability distributions, the system achieves more reliable anomaly detection with fewer false positives while maintaining detection speed.
2Productivity
If high-dimensional data is processed using traditional methods, then data complexity is handled, but processing efficiency decreases
Solution Approach 1:
The patent merges multiple anomaly detection operations into a single KL divergence computation. By combining the comparison of multiple probability distributions into one unified operation, the system efficiently handles high-dimensional data without sacrificing processing efficiency or increasing computational complexity.
3Reliability
If correlation patterns are not considered, then detection simplicity is maintained, but anomaly detection capability deteriorates
Solution Approach 1:
The KL divergence computation serves multiple functions simultaneously: it compares probability distributions, detects anomalies, identifies correlation patterns, and handles high-dimensional data all in one operation. This multi-functionality enables the system to detect anomalies based on correlation patterns without adding separate complex correlation analysis mechanisms.
Data Source
AI summary
An embodiment establishes a system behavior database based at least in part on behavior data received from a system, wherein the system behavior database comprises a set of historical observations of behavior of the system. The embodiment samples current behavior data from the system. The embodiment generates a current observation based on the current behavior data that was sampled. The embodiment compares the current observation to each historical observation of the set of historical observations to determine a divergence between the current observation and each historical observation. The embodiment compares the divergence to a divergence threshold, and upon a determination that the divergence exceeds the divergence threshold, detects an anomaly in the system, updates the set of historical observations to include the current observation as a new historical observation, and performs a responsive action within the system based on the detected anomaly.