Cloud Metric Data Management via Sequential Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of metric data generated by cloud-computing infrastructures makes it challenging for IT managers to identify and isolate abnormalities, as existing methods struggle to efficiently manage and analyze large volumes of 'big data' effectively.
Innovation Solution
The implementation of a system with multiple modules that apply sequential data reduction techniques, including metric reduction, normalcy analysis, and anomaly detection, to filter out redundant and correlated data, determine normalcy bounds, and rank abnormalities, thereby reducing data complexity and identifying root causes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud-computing infrastructures generate and store millions of different types of metrics over time, then the volume of metric data increases, but the ability to identify and isolate abnormalities becomes increasingly challenging
Solution Approach 1:
The patent segments the large volume of metric data into smaller, manageable groups based on correlations. It identifies and isolates subsets of metrics that are highly correlated with abnormalities, separating them from the rest of the data. This segmentation allows IT managers to focus on specific groups of metrics rather than analyzing all millions of metrics, thereby reducing the difficulty of identifying abnormalities while preserving the necessary data volume for comprehensive monitoring.
Solution Approach 2:
The patent extracts and removes highly correlated metric data from the large dataset. By identifying metrics that are strongly correlated with abnormalities and extracting them into a separate analysis group, the system reduces the overall data volume that needs to be processed for anomaly detection. This extraction principle allows the system to work with a smaller, more manageable subset of metrics while maintaining the ability to detect abnormalities effectively.
2Quantity of substance
If sequential data reduction techniques are applied to reduce metric data volume, then the volume of metric data is reduced and learning accuracy is increased, but the complexity of the data management system increases
Solution Approach 1:
The patent applies preliminary actions by pre-calculating and storing correlation information between metrics before actual anomaly detection is needed. The system pre-identifies which metrics are highly correlated with each other and with abnormalities, creating a structured representation of these relationships in advance. This preliminary action reduces the computational complexity during runtime, as the system can directly query pre-computed correlation data rather than performing complex analyses on the full dataset when abnormalities occur.
Solution Approach 2:
The patent introduces intermediary structures such as correlation matrices and metric groupings that mediate between the raw metric data and the anomaly detection process. These intermediaries organize and summarize the relationships between metrics, serving as a bridge that simplifies the complexity of analyzing millions of individual metrics. The intermediary structures enable the system to manage data reduction while maintaining analytical accuracy without directly processing the full complexity of the original dataset.
Data Source
AI summary
Methods and systems that manage large volumes of metric data generation by cloud-computing infrastructures are described. The cloud-computing infrastructure generates sets of metric data, each set of metric data may represent usage or performance of an application or application module run by the cloud-computing infrastructure or may represent use or performance of cloud-computing resources used by the applications. The metric data management methods and systems are composed of separate modules that perform sequential application of metric data reduction techniques on different levels of data abstraction in order to reduce volume of metric data collected. In particular, the modules determine normalcy bounds, delete highly correlated metric data, and delete metric data with highly correlated normalcy bound violations.