Dynamic Window Duration for Anomaly Detection in Cloud Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional anomaly detection systems in cloud applications use fixed window durations for log data analysis, which can miss anomalies with signatures spread across multiple windows or buried in small portions, as they do not adapt to changing system activity levels.
Innovation Solution
The system dynamically adjusts the anomaly detection window duration based on activity scores calculated from system metrics and causal relationships, reducing the window duration during high activity and increasing it during low activity to enhance anomaly detection sensitivity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a fixed window duration is used for anomaly detection, then the system structure is simple and easy to implement, but the detection sensitivity decreases when system activity levels change
Solution Approach 1:
The patent implements dynamic window size adjustment by continuously monitoring system activity metrics and adapting the analysis window duration accordingly. The window size changes from a fixed value to a dynamic parameter that responds to system conditions, allowing the anomaly detection system to maintain high sensitivity across varying activity levels without requiring multiple static configurations.
Solution Approach 2:
The core mechanism involves changing the time window parameter based on system activity metrics. When activity levels change, the window duration is adjusted to optimize detection sensitivity. This parameter adaptation allows the system to handle both high-activity and low-activity scenarios effectively, resolving the contradiction between adaptability and complexity by focusing on a single adjustable parameter rather than multiple system components.
2Speed
If a short window duration is used, then short duration anomalies can be detected quickly, but anomalies with signatures spread across multiple windows may be missed
Solution Approach 1:
The system dynamically adjusts window duration based on detected anomaly characteristics. When an anomaly signature is detected spanning multiple windows, the system extends the window duration to capture the complete anomaly pattern, ensuring reliable detection without sacrificing the initial quick detection capability of shorter windows.
Solution Approach 2:
The patent merges multiple short-time windows into a longer analysis window when anomaly signatures indicate a spread pattern. This combining approach allows the system to maintain fast initial detection with short windows while ensuring complete anomaly capture by extending to longer windows when needed, thus resolving the contradiction between speed and reliability.
3Reliability
If a long window duration is used, then anomalies with spread signatures can be captured completely, but the detection response time increases and short duration anomalies may be missed
Solution Approach 1:
The system uses dynamic window adjustment to avoid the constant overhead of long windows. When system activity indicates stable conditions, shorter windows provide fast response. When anomaly patterns suggest spread signatures, the window extends to capture the complete pattern. This dynamic behavior eliminates unnecessary time loss from using long windows during stable periods while maintaining reliability when needed.
Solution Approach 2:
The patent implements periodic evaluation of system activity metrics to determine appropriate window duration. This periodic adjustment allows the system to switch between short and long windows based on current conditions, maintaining fast response during normal operation while ensuring complete anomaly capture when patterns indicate spread signatures, thus resolving the time loss versus reliability contradiction.
Data Source
AI summary
Detecting system log anomalies by receiving multivariate time-series system log data an multivariate metric data, inferring system metrics from the system log data and metric data, receiving a metric causal graph including causal relationships between system metrics, determining a univariate variation score for the system metrics, determining a causal variation score for the multivariate time series system metric data, according to the causal graph, determining an overall activity score according to the univariate variation score, and causal variation score, and altering a review window duration according to the activity score.