Run Index Compression for Time-Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for compressing run indexes in time-series data are inefficient, leading to high memory utilization and slow query performance due to the large number of entries in uncompressed run indexes, especially in scenarios with bulk-loaded data containing multiple runs of varying lengths.
Innovation Solution
The proposed solution involves analyzing the run index and determining the most suitable compression method based on memory consumption, applying vertical and horizontal compression techniques such as dynamic ranging, constant ranging, linear run length encoding, and checkpoint compression to reduce the number of entries in the run index while maintaining rapid lookup capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If uncompressed run index is used, then query lookup capability is maintained, but memory utilization increases significantly
Solution Approach 1:
The run index is divided into multiple partitions, each storing run index values for a specific range of runs. This segmentation allows the index to be stored in a compressed format while maintaining efficient lookup capabilities through partition-specific optimization.
Solution Approach 2:
The patent implements a nested structure where run index values are organized hierarchically within partitions. Each partition contains compressed representations of run data, with offset columns enabling navigation through nested levels of compression, effectively reducing memory usage while preserving access capability.
2Quantity of substance
If multiple compression methods are applied to run index, then memory utilization is reduced, but device complexity increases
Solution Approach 1:
The patent dynamically selects and applies different compression methods (vertical compression, horizontal compression, or no compression) to different partitions of the run index based on the specific characteristics of each partition. This dynamic approach optimizes memory usage while managing complexity through selective application.
Solution Approach 2:
The system changes compression parameters by applying different compression techniques to different portions of the run index. Vertical compression modifies the structure along the value dimension, while horizontal compression operates along the run dimension, allowing flexible parameter adjustment to balance memory usage and complexity.
3Quantity of substance
If run index is partitioned and compressed, then storage efficiency is improved, but query processing complexity increases
Solution Approach 1:
The run index is pre-partitioned and compressed into multiple organized partitions with offset columns before query processing. This preliminary organization enables efficient query execution by allowing the system to quickly locate relevant partitions and apply appropriate decompression only when needed, rather than processing the entire index.
Data Source
AI summary
A database can receive a query from a remote computing system. The database can include (i) a linear run length encoded compressed column, based on an original column of time series data partitioned into runs containing consecutive values and generated by run length encoding, (ii) a run index comprising at least one run index value having a run index position, the at least one run index value identifying runs in the original column, and (iii) an offsets column identifying the run index positions corresponding to the runs that contain a desired value. Using the run index, data responsive to the query can be identified. The identified data responsive to the query can be transmitted by the database to the remote computing system.