Timestamp Column Compression via Parametric Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in optimizing memory consumption and performance when handling series data, particularly time stamps, due to linear memory usage and lack of support for gaps in time series data.
Innovation Solution
The proposed solution involves compressing time stamp columns by storing only the start time and interval in a dictionary, generating index vectors, and calculating time stamps based on these, which allows for constant memory consumption and supports gaps by dividing time stamps into blocks with additional vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If time stamps are stored in a traditional column store, then memory consumption is linear with the number of time stamps, but performance is improved with direct access
Solution Approach 1:
The patent transforms the storage representation of time stamps from absolute values to a parametric form using start time and interval. Instead of storing each time stamp value, the system stores only the start time and the interval between successive time stamps, allowing reconstruction of any time stamp through calculation. This parameter transformation reduces memory consumption from linear O(n) to constant O(1) while maintaining performance through direct calculation access.
Solution Approach 2:
The patent creates a compressed representation (copy) of the time stamp data that uses significantly less memory. Instead of copying and storing all time stamp values, it stores only the essential parameters (start time and interval) that can generate any time stamp in the series, effectively creating a compact copy that preserves all necessary information while minimizing storage requirements.
2Quantity of substance
If time stamps are compressed to reduce memory usage, then memory consumption becomes constant, but handling gaps in time series becomes complex
Solution Approach 1:
The patent segments the time series data into multiple blocks, where each block represents a continuous sequence of time stamps without gaps. By dividing the overall time series into discrete blocks, the system can independently store and manage each block's start time and interval, making it easier to handle gaps between blocks while maintaining the compression benefits within each block.
Solution Approach 2:
The patent introduces an intermediary structure (block metadata and indexing system) that manages the segmented time series blocks. This intermediary layer handles the complexity of gaps by providing a structured way to represent, store, and access discontinuous time series data, shielding the compression mechanism from the complexity of gap handling while enabling efficient query processing across block boundaries.
Data Source
AI summary
Disclosed is a system and method for improving database memory consumption and performance using compression of time stamp columns. A number of time stamps of a time series is received. The time stamps have a start time, and are separated by an equal increment of time that defines an interval. The start time and interval are stored in a dictionary of a column store of a database. An index is generated in the column store of the database, the index having a number of index vectors. Using the index vectors, each time stamp of the number of time stamps can be calculated from the start time stored in the dictionary and the position in the time series based on the interval stored in the dictionary.