Columnar Storage for Time Series Data with Page-Level Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current columnar file formats, such as Parquet and ORCFile, do not optimize for time series data, leading to slow query speeds and increased system resource consumption due to the inability to store timestamps and data values in one column simultaneously, which complicates data storage and retrieval.
Innovation Solution
A columnar storage method that divides time series data into pages with a page header containing summary index information and a page body storing data values, allowing for efficient storage and querying by encoding timestamps and data values separately and using summary index information to filter query results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If time series data is stored in conventional columnar formats (Parquet, ORCFile), then data can be stored in columnar structure, but timestamps and data values cannot be stored in one column simultaneously, resulting in complicated nested structure and slow query speed
Solution Approach 1:
The patent segments time series data into multiple pages, where each page contains a fixed number of data points with their corresponding timestamps and values. This segmentation allows the data to be organized in a simplified columnar structure where timestamps, data values, and other attributes can be stored in separate columns within each page, eliminating the need for complicated nested structures while enabling efficient querying through page-level operations.
2Quantity of substance
If conventional columnar formats are used for time series data, then storage is possible, but I/O operations increase and system resources are consumed
Solution Approach 1:
The patent performs preliminary actions by pre-organizing time series data into fixed-size pages during the storage phase. Each page is pre-computed with a specific number of data points and their corresponding metadata. This preliminary organization enables efficient I/O operations during querying, as the system can read entire pages at once rather than performing multiple smaller I/O operations, thereby reducing overall system resource consumption while maintaining storage capacity.
3Productivity
If data points are divided into multiple pages, then query efficiency can be improved through selective reading, but data storage space requirements increase
Solution Approach 1:
The patent applies parameter changes by implementing a fixed page size parameter for organizing time series data. Each page contains a predetermined number of data points (e.g., 4096 points per page), which optimizes the balance between query efficiency and storage space utilization. This parameter-based organization allows the system to read only the necessary pages during querying, improving retrieval efficiency while the fixed-size structure prevents excessive storage overhead through efficient memory management and compression techniques.
Data Source
AI summary
A columnar storage method and a query method and system for time series data. The storage method includes: dividing a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data (S1); and setting two parts, i.e., a page header and a page body, for each page, storing summary index information of all the data points in the page in the page header of the page and storing data value information of all the data points in the page in the page body of the page (S2).