Telemetry Time-Series Compression for Bounded Historical Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise computing systems face challenges in archiving historical performance data due to limited storage space, leading to the loss of data older than a certain period when using fixed-size buffers, which hinders long-term monitoring and fault diagnosis.
Innovation Solution
A system with three buffers (two current data buffers and one historical data buffer) that compresses telemetry data by creating summary statistics, allowing for efficient storage of both short-term and long-term performance data, with the historical buffer containing sets of summary statistics with varying levels of compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If real-time performance data is stored continuously in a fixed-size buffer, then storage capacity is efficiently utilized, but historical data older than the buffer period is permanently lost
Solution Approach 1:
The system dynamically adjusts the storage period for different data by identifying inflection points in the time series data. Recent data is stored with shorter periods while historical data is stored with longer periods, allowing the buffer to adapt its retention strategy based on data characteristics rather than using a fixed period for all data.
Solution Approach 2:
The system changes the storage period parameter based on the identified inflection points in the data. By detecting significant changes in system behavior, the system adjusts how long different segments of data are retained, extending retention for critical historical periods while maintaining efficient storage utilization.
2Loss of information
If the buffer size is increased to retain more historical data, then data retention period is extended, but storage space is wasted on low-value data
Solution Approach 1:
The time series data is segmented into multiple periods based on inflection points, with each segment having a different retention duration. This allows the system to retain detailed information for critical periods while using coarser representations for less critical historical periods, optimizing storage efficiency while maintaining data retention for important events.
Solution Approach 2:
Different retention qualities are applied to different segments of the time series data. Recent and critical historical data near inflection points receive higher retention priority with shorter periods, while older or less critical data uses longer periods with reduced detail, ensuring storage space is allocated according to data value rather than uniformly.
3Measurement precision
If detailed real-time data is archived for long-term monitoring, then monitoring precision is improved, but storage requirements exceed available capacity
Solution Approach 1:
The system dynamically adjusts the level of detail stored based on the time period and data characteristics. Near inflection points, detailed real-time data is preserved for high-precision monitoring, while in stable periods, coarser summaries are stored, reducing storage requirements while maintaining the ability to detect significant events.
Solution Approach 2:
The storage period parameter is changed based on identified inflection points in the data. By detecting significant changes in system behavior, the system adjusts the monitoring precision and data retention strategy for different time segments, maintaining high precision where needed while reducing storage requirements in stable periods.
Data Source
AI summary
Some embodiments of the present invention provide a system that stores telemetry data from a computer system. The system includes a first buffer, a second buffer, and a third buffer. During operation, the system periodically obtains the telemetry data from the computer system and stores the telemetry data in the first buffer, second buffer, and third buffer. The system also compresses the telemetry data in the first and second buffers. To compress the data, the system creates a first set of summary statistics from the telemetry data in the first buffer and the second buffer and stores the first set of summary statistics in the first buffer, which becomes a historical data buffer.