Linear Run Length Encoding for Time Series Index Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in optimizing memory consumption and performance when handling series data, particularly with time series analysis, as existing compression methods do not efficiently handle equidistant time series data with gaps, leading to increased memory usage and complex operations.
Innovation Solution
The implementation of linear run length encoding (LRLE) for compressing index vectors in time series data, which stores only the starting position and starting value for each block, allowing for efficient calculation of run lengths and supporting gapless and gap-containing sequences, thereby optimizing memory usage and access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression methods are used for time series data, then memory consumption is reduced, but access efficiency and performance deteriorate
Solution Approach 1:
The time series data is divided into blocks, where each block represents a contiguous sequence of time intervals. The index vector is segmented to store only block-level information (starting position and starting value) rather than individual time stamp information, reducing memory consumption while maintaining efficient access through block-based retrieval.
Solution Approach 2:
The patent extracts and stores only the essential information needed for reconstruction - specifically the starting position and starting value of each block. Redundant information such as individual time stamps within blocks is removed, as these can be regenerated from the block metadata and the equidistant time interval property.
2Measurement precision
If detailed index information is stored for each time interval, then access precision is improved, but memory usage increases
Solution Approach 1:
The patent pre-calculates and stores the starting position and starting value for each block before access occurs. This preliminary action enables rapid reconstruction of individual time interval values during access operations, maintaining precision without storing all intermediate values in memory.
Solution Approach 2:
The system uses the equidistant time interval property to enable self-service reconstruction of time stamp values. Given the block's starting position, starting value, and the known equidistant increment, any individual time interval value within the block can be regenerated on-demand without storing it explicitly.
3Reliability
If the entire index vector is recreated when appending values, then data consistency is maintained, but operation complexity and time increase
Solution Approach 1:
The index vector is organized as a collection of independent blocks. When appending new values, only the necessary block metadata (starting position and starting value) needs to be added or updated, rather than recreating the entire index vector. This segmentation enables incremental updates while maintaining consistency.
Solution Approach 2:
The block structure allows preliminary preparation of new blocks with their starting position and starting value before actual data appending. This pre-organization enables seamless integration with existing blocks without requiring complex full-vector reconstruction operations.
Data Source
AI summary
A system and method include storing a table of time series data in a database of a data platform, the table of time series data representing a set of time series blocks. Each time series block of the set of time series blocks has a time series of equally-incremented time intervals and a run length. Each time interval of the time series is associated with one or more values. The run length has a starting position with at least one starting value and an ending position with at least one ending value. The starting position and the at least one starting value is stored for each time series block in a column store of the database. Then, a compressed index is generated in the column store of the database for each time series block, the compressed index comprising the starting position and the at least one starting value.