Dynamic Threshold Anomaly Detection in Data Centers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods in data centers rely on static thresholds, which are invariant to changes in workload characteristics over time, leading to reduced accuracy and increased false alarms, and are not scalable for complex cloud computing environments.
Innovation Solution
A windowing-based approach that adapts to changing workload characteristics by computing dynamic thresholds based on historical data, using non-parametric methods like Tukey's method and multinomial goodness-of-fit tests to identify anomalies without assuming a specific distribution, and applying these methods in an online, closed-loop system for continuous monitoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If static thresholds are used for anomaly detection, then the system is simple to implement, but accuracy decreases and false alarms increase due to inability to adapt to changing workload characteristics
Solution Approach 1:
The patent transforms static thresholds into dynamic thresholds that automatically adapt to changing workload characteristics. The system continuously updates thresholds based on recent historical data, allowing them to respond to seasonal patterns, workload bursts, and other temporal variations in data center behavior, thereby resolving the contradiction between implementation simplicity and detection accuracy.
Solution Approach 2:
The patent changes the parameter of threshold values from fixed to variable based on time and workload conditions. By computing thresholds as functions of time and historical data patterns, the system adjusts threshold parameters dynamically to match current operational characteristics, eliminating false alarms while maintaining ease of implementation through automated threshold computation.
2Measurement precision
If academic statistical techniques are used for anomaly detection, then detection accuracy may improve, but computing overhead increases and scalability to large data centers becomes difficult
Solution Approach 1:
The patent extracts only the essential components needed for effective anomaly detection from complex academic statistical methods. By focusing on threshold computation based on recent historical data patterns rather than comprehensive statistical analysis, the system achieves sufficient accuracy with significantly reduced computing overhead, making it scalable to large data centers.
Solution Approach 2:
The patent applies partial statistical analysis focusing on recent historical data patterns rather than complete statistical treatment. This selective approach computes thresholds based on a manageable portion of historical data, providing adequate detection accuracy while avoiding the excessive computational requirements of full academic statistical techniques.
3Adaptability or versatility
If MASF with time-segmented thresholds is used, then adaptability to temporal patterns improves, but the Gaussian distribution assumption is frequently violated in practice reducing effectiveness
Solution Approach 1:
The patent changes the underlying assumption from Gaussian distribution to non-parametric threshold computation based on recent historical data patterns. This allows the system to adapt to temporal patterns while remaining effective under any data distribution, eliminating the reliability issues associated with MASF's Gaussian assumption.
Solution Approach 2:
Instead of assuming a specific distribution (Gaussian) and deriving thresholds from that assumption, the patent inverts the approach by directly computing thresholds from observed historical data patterns. This data-driven inversion eliminates distributional assumptions and maintains effectiveness regardless of whether data follows Gaussian or non-Gaussian distributions.
4Device complexity
If fixed thresholds are used, then system complexity is low, but the system cannot respond to intermittent bursts or workloads that change in nature over time
Solution Approach 1:
The patent introduces dynamic threshold computation that automatically adapts to changing workload characteristics without requiring complex manual configuration. The system continuously learns from historical data and adjusts thresholds in response to intermittent bursts and workload changes, achieving adaptability while maintaining relatively low system complexity through automated computation.
Data Source
AI summary
Systems and methods of anomaly detection in data centers. An example method may include analyzing time series data for the data center by testing statistical hypotheses. The method may also include constructing upper and lower bounds based on the statistical hypotheses. The method may also include flagging anomalies in the time series data falling outside of the upper and lower bounds.