Block Index for Compressed Time Series Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems face challenges in optimizing memory consumption and performance when handling time series data, particularly in supporting equidistant time series with gaps, which leads to increased memory usage and access times.
Innovation Solution
The implementation of a block index system that compresses index vectors by dividing time stamps into blocks, using a start vector and a starting value vector to represent block positions and values, and applying linear run length encoding (LRLE) to reduce memory consumption and enhance access efficiency, while supporting gaps in time series data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If time series data is stored with full index vectors for each timestamp, then data access precision is improved, but memory consumption increases
Solution Approach 1:
The patent divides the time series data into blocks where each block contains a range of consecutive timestamps. Instead of storing full index vectors for every timestamp, the patent stores compressed block indices that represent ranges of timestamps. This segmentation allows precise access to specific time ranges while reducing memory consumption by not storing redundant index information for every individual timestamp.
Solution Approach 2:
The patent transforms the storage representation from storing complete index vectors for each timestamp to storing compressed block indices with start positions and length information. This parameter change in the data structure allows the system to maintain access precision by storing block-level metadata while significantly reducing memory consumption by eliminating redundant timestamp index information.
2Quantity of substance
If compressed block indices are used to reduce memory consumption, then memory efficiency is improved, but data access speed may worsen
Solution Approach 1:
The patent pre-organizes data into blocks with stored start positions and length information during the data ingestion phase. This preliminary action of structuring data into manageable blocks with metadata allows for efficient querying later, as the system can quickly locate relevant blocks without scanning entire datasets, thus maintaining fast access speeds despite compression.
Solution Approach 2:
The patent introduces block indices as an intermediary layer between the raw time series data and query operations. These block indices act as a metadata layer that stores start positions and length information, enabling the query system to efficiently locate and retrieve specific time ranges without directly processing compressed data, thus maintaining access speed while benefiting from compression.
3Ease of operation
If traditional indexing methods are used for time series data, then query simplicity is maintained, but memory consumption increases
Solution Approach 1:
The patent segments time series data into blocks with associated metadata (start position, length), creating a hierarchical structure that maintains query simplicity through block-level operations while reducing memory consumption by not storing full index vectors for every timestamp. Queries can operate on blocks rather than individual timestamps, simplifying the indexing structure.
4Adaptability or versatility
If gaps in time series data are supported, then data flexibility is improved, but memory usage increases
Solution Approach 1:
The patent uses variable-length block encoding where each block stores start position and length parameters. This parameter change allows blocks to dynamically represent continuous time ranges or handle gaps by simply adjusting the length field, providing data flexibility to support gaps without requiring additional memory structures or complex encoding schemes.
Data Source
AI summary
A system and method for faster access for compressed time series data. A set of blocks are generated based on a table stored in a database of the data platform. The table stores data associated with multiple sources of data provided as consecutive values, each block containing index vectors having a range of the consecutive values. A block index is generated for each block having a field start vector representing a starting position of the block relative to the range of consecutive values, and a starting value vector representing a value of the block at the starting position. The field start vector of the block index is accessed to obtain the starting position of a field corresponding to a first block and to the range of the consecutive values of the first block. The starting value vector is then determined from the block index to determine an end and a length of the field of the first block.