Anomalous Behavior Detector Using Dynamic Statistical Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing diverse data systems to detect anomalous behavior, such as malicious activity in cloud computing services, is challenging due to conventional detectors often being too sensitive or not sensitive enough, failing to provide meaningful insights.
Innovation Solution
An event-based data intake and query system with a late-binding schema that processes and indexes machine data, allowing for flexible schema definition and extraction rules application at search time, enabling effective detection of anomalous behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional detectors use fixed thresholds for anomaly detection, then the detection process is simple and fast, but the detectors become either too sensitive or not sensitive enough, failing to provide meaningful insights
Solution Approach 1:
The patent applies dynamics by transitioning from fixed static thresholds to dynamic thresholds that adapt automatically based on statistical modeling of the data. The system continuously learns from observed data patterns and adjusts detection thresholds accordingly, enabling the detector to maintain optimal sensitivity and specificity over time without manual reconfiguration.
Solution Approach 2:
The patent changes the parameter of detection thresholds from fixed values to dynamically adjusted values based on statistical distributions. By modeling the statistical characteristics of normal behavior and using these models to determine adaptive thresholds, the system achieves both speed and accuracy in anomaly detection.
2Ease of manufacture
If detectors use fixed thresholds configured manually, then the configuration process is straightforward, but the detectors cannot adapt to changing data patterns and become less effective over time
Solution Approach 1:
The patent implements self-service by enabling the detector to automatically configure and adjust its own thresholds based on observed data patterns. Instead of requiring manual configuration that becomes obsolete, the system performs self-learning and self-adjustment, continuously adapting to changing data distributions while maintaining ease of initial setup through automated threshold generation.
Solution Approach 2:
The system incorporates feedback mechanisms where detected anomalies and data patterns are fed back into the statistical model to refine future detection thresholds. This continuous feedback loop enables the detector to adapt to evolving data characteristics while maintaining straightforward initial configuration through automated learning from historical data.
3Adaptability or versatility
If the system stores and analyzes all raw data, then the flexibility to analyze different aspects of data is maximized, but the complexity of managing and searching through vast amounts of data increases
Solution Approach 1:
The patent applies extraction by identifying and isolating the statistical characteristics and patterns from the vast raw data. Instead of managing all raw data uniformly, the system extracts key statistical features, distributions, and patterns, storing and managing only these condensed representations. This enables flexible analysis of different data aspects while dramatically reducing management complexity through dimensionality reduction.
Solution Approach 2:
The system segments the data management approach by separating raw data storage from statistical model storage. Raw data is maintained for reference, while extracted statistical characteristics are stored separately in optimized structures. This segmentation allows flexible querying of different data aspects through the statistical models without requiring full scans of all raw data, reducing operational complexity.
Data Source
AI summary
A computer-implemented method of configuring an anomalous behavior detector includes updating a distribution used for modeling anomalous behavior in telemetry data with information associated with observed anomalous behavior to generate an updated distribution representative of the observed anomalous behavior where, prior to the updating, the distribution is representative of theoretical anomalous behavior. The method further includes computing a threshold for a detector operable to alert on anomalous activity using the updated distribution. The method also comprises computing a divergence between the live telemetry data monitored by the detector and the anomalous behavior modeled by the updated distribution. Responsive to a determination that the divergence is above a critical threshold, the method comprises enabling the detector to continue to monitor the live telemetry data in the application.