Floating-Point Time-Series Compression Using Offset Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing compression techniques for floating point numbers in time-series data are inefficient due to the difficulty in compressing data formats like IEEE 754, which results in high overhead and low throughput in high performance computing environments, especially when dealing with spatially and temporally continuous data.
Innovation Solution
A spatial-temporal compression technique that leverages both spatial and temporal information by using an offset predictor to enhance prediction accuracy, reducing prediction errors, and thereby achieving higher compression ratios with improved throughput by reusing the best predictor for subsequent checkpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If standard compression techniques are used for floating point numbers, then data can be compressed, but the compression ratio is low and processing overhead is high
Solution Approach 1:
The patent applies preliminary action by performing prediction on floating point values before compression. The offset predictor estimates future values based on historical data, and only the prediction error (difference between actual and predicted values) is compressed and stored. This preliminary prediction step transforms the data into a form that requires significantly less storage space and can be processed more efficiently.
Solution Approach 2:
The patent changes the parameter representation by transforming floating point values into prediction errors. Instead of compressing the original floating point numbers directly, the system transforms them into differential values that represent deviations from predicted values. This parameter transformation enables much higher compression ratios because the prediction errors are typically small and can be encoded more efficiently.
2Loss of substance
If compression ratio is increased, then storage efficiency improves, but processing complexity increases
Solution Approach 1:
The patent applies self-service by using the data's own historical patterns to predict future values. The offset predictor learns from the temporal and spatial characteristics inherent in the time-series data itself, using the data's own structure to generate predictions. This eliminates the need for complex external compression algorithms and reduces processing overhead while achieving high compression ratios.
Solution Approach 2:
The patent implements feedback by using the prediction error to adjust and refine subsequent predictions. The system continuously monitors the difference between predicted and actual values, and this feedback information is used to improve the accuracy of future predictions. This feedback mechanism enables the system to adapt to changing data patterns while maintaining computational efficiency.
3Measurement precision
If prediction accuracy is improved, then compression ratio increases, but computational resources required increase
Solution Approach 1:
The patent applies segmentation by dividing the prediction task into spatial and temporal components. The offset predictor separately analyzes spatial patterns (across different data dimensions) and temporal patterns (across time steps), combining these segmented analyses to generate predictions. This segmentation allows the system to achieve high prediction accuracy without requiring computationally intensive holistic analysis of the entire dataset.
Solution Approach 2:
The patent applies partial action by focusing computational resources on predicting only the most significant components of the data. Rather than attempting to predict every detail with equal precision, the offset predictor concentrates on capturing the dominant temporal and spatial trends, accepting that minor variations may not be perfectly predicted. This selective approach achieves sufficient accuracy for compression purposes while conserving computational resources.
Data Source
AI summary
Computer-implemented methods, systems, and devices to perform lossless compression of floating point format time-series data are disclosed. A first data value may be obtained in floating point format representative of an initial time-series parameter. For example, an output checkpoint of a computer simulation of a real-world event such as weather prediction or nuclear reaction simulation. A first predicted value may be determined representing the parameter at a first checkpoint time. A second data value may be obtained from the simulation. A prediction error may be calculated. Another predicted value may be generated for a next point in time and may be adjusted by the previously determined prediction error (e.g., to increase accuracy of the subsequent prediction). When a third data value is obtained, the adjusted prediction value may be used to generate a difference (e.g., XOR) for storing in a compressed data store to represent the third data value.