Metric Data Compression for Peak Detection in Computing Facilities
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer system monitoring and management tools face challenges in efficiently collecting, storing, and analyzing large volumes of metric data from distributed computing facilities, leading to significant storage and computational overheads, and difficulties in identifying long-term patterns due to the expense of storing uncompressed data.
Innovation Solution
The method involves compressing metric data points into one-bit, two-bit, four-bit, or eight-bit values, allowing for immediate analysis in uncompressed form, followed by compression and storage in memory, and eventual archiving in mass storage, enabling detection of peaks and patterns over extended periods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If metric data is stored in uncompressed form for immediate analysis, then data analysis and monitoring functionalities can operate efficiently, but storage requirements and computational overheads become significant
Solution Approach 1:
The patent segments metric data into different storage locations based on their usage requirements: recent uncompressed data is kept in memory for immediate analysis, while historical compressed data is archived in mass storage. This segmentation allows the system to maintain fast access to recent data while reducing overall storage requirements through compression of older data.
Solution Approach 2:
The system dynamically switches between uncompressed and compressed data representations based on the time window requirements. Recent data within the first time window is accessed in uncompressed form for immediate analysis, while data beyond this window is accessed in compressed form from mass storage, enabling the system to adapt its data representation based on operational needs.
2Measurement precision
If uncompressed metric data is retained for long-term analysis, then pattern detection accuracy improves, but storage costs and computational burdens increase significantly
Solution Approach 1:
The patent creates a compressed copy of metric data that preserves the essential information needed for pattern detection while reducing the data volume. The compressed data structure maintains the temporal and value information required for detecting peaks and periodic patterns, allowing long-term retention without the full computational burden of uncompressed data.
3Quantity of substance
If metric data is compressed immediately upon reception, then storage efficiency improves, but data availability for immediate analysis is reduced
Solution Approach 1:
The system performs preliminary compression of metric data immediately upon reception and storage in mass storage, but maintains a separate uncompressed buffer for the first time window period. This preliminary action prepares data for long-term efficient storage while ensuring immediate availability for analysis within the short-term window before compression takes effect.
Data Source
AI summary
The current document is directed to methods and systems that collect metric data within computing facilities, including large data centers and cloud-computing facilities. In a described implementation, input metric data is compressed by replacing each metric data point with a one-bit, two-bit, four-bit, or eight-bit compressed data value. During a first time window following reception of a metric data point, the metric data point remains available in uncompressed form to facilitate data analysis and monitoring functionalities that use uncompressed metric data. During a second time window, the metric data point is compressed and stored in memory, where the compressed data point remains available for data analysis and monitoring functionalities that use compressed metric data for detection of peaks, periodic patterns, and other characteristics. Finally, the compressed data point is archived in mass storage, where it remains available to data-analysis and management functionalities for a lengthy time period.