Cloud Metric Filtering with Feedback Control for Data Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The sheer volume of metric data generated by cloud applications makes analysis and storage of all available metric values intractable, as existing systems struggle to distinguish between normal and abnormal behavior, leading to inefficient data storage and performance degradation.
Innovation Solution
A cloud monitoring system employs a filtering paradigm using stateless and stateful time-series filters, including a persistence/coherence filter and a cloud metric filter with a feedback control loop, to identify and eliminate non-extremal metric values, reducing data storage volume by orders of magnitude and focusing on statistically significant metric data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all metric data is stored and analyzed, then complete observation of system behavior is achieved, but storage volume and processing complexity become intractable
Solution Approach 1:
The patent extracts only the extremal metric values (maximum and minimum) from the complete metric data stream, discarding non-extremal values. This extraction principle reduces storage volume by focusing only on the most significant data points that indicate abnormal system behavior, while still maintaining the ability to detect anomalies.
Solution Approach 2:
The patent applies different quality treatment to different portions of metric data by identifying and preserving only the extremal values (local quality characteristics) while discarding the bulk of normal data. This allows the system to maintain high-quality anomaly detection capability with minimal storage requirements.
2Quantity of substance
If extremal metric values are tracked to reduce storage volume, then data storage efficiency is improved, but false positives from normal extremal behavior increase
Solution Approach 1:
The patent dynamically adjusts the baseline metric values and thresholds based on historical data and changing system conditions. By making the detection criteria adaptive rather than static, the system can distinguish between normal extremal behavior (such as expected peak loads) and abnormal behavior, reducing false positives while maintaining sensitivity to real anomalies.
Solution Approach 2:
The patent implements a feedback mechanism where detected extremal values are used to update baseline expectations and adjust future detection thresholds. This feedback loop allows the system to learn from normal extremal patterns and improve its ability to distinguish between normal and abnormal behavior over time, enhancing detection reliability.
3Productivity
If filtering is applied to reduce data volume, then processing performance is enhanced, but complexity of filter configuration and maintenance increases
Solution Approach 1:
The patent segments the filtering process into distinct, modular components: an extremal value identification filter that selects maximum and minimum values, and a persistence filter that checks for continuity across time windows. This segmentation allows each filter to be independently configured, maintained, and optimized, reducing overall system complexity while maintaining effective filtering capability.
Data Source
AI summary
A cloud monitoring system is disclosed herein that uses a filtering paradigm after metric data aggregation and before storing in a repository that allows querying the metric data to significantly reduce the raw data stored into the repository. A persistence filter identifies extremal cloud metrics that are persistent across time windows, increasing the confidence that extremal metrics correspond to abnormal behavior. A cloud metric filter comprising a sequence of logical and statistical filters that are interchangeable in order and use allows for dynamic filtering of cloud data. The cloud metric filter has a feedback control loop to update the order and parameters of individual filters based on properties of filtered metric values. Intelligent filtering of cloud metric data yields a focused set of statistically significant metric data for monitoring and eliminates noisy metric data for normal behavior.