Formula-Encoded Timestamps for Time Series Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The storage of time-stamped data in database management systems, particularly in high-speed in-memory databases, is inefficient due to the large storage requirements for timestamps, which can lead to high memory usage and limitations in operations.
Innovation Solution
Physically sorting rows to group timestamp values encodable by a linear formula, computing and storing the increment and offset, and applying run-length compression to these values, allowing timestamp values to be represented by a single slope and offset, with optional jitter values to account for deviations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If time stamp values are stored in a database table, then the time series data can be stored and retrieved, but the data size becomes extremely high requiring substantial storage space and main system memory
Solution Approach 1:
The patent changes the parameter representation of time stamp data from storing actual timestamp values to storing encoding parameters (start value, increment, offset). This transforms the data from requiring 4-6 bytes per timestamp to requiring only 3 parameters per group, dramatically reducing storage requirements while maintaining the ability to retrieve and use the time series data
Solution Approach 2:
The patent segments the time series data into groups where each group shares common encoding parameters (start value, increment, offset). By dividing the data into segments that can be independently encoded, the system reduces overall storage requirements while maintaining data integrity and accessibility for query operations
2Quantity of substance
If compression methods are employed to reduce storage requirements for measurement columns, then storage space is reduced, but such approaches are not as useful for time stamp data because individual data values are unique or nearly unique
Solution Approach 1:
The patent applies parameter changes specifically to time stamp data by representing each timestamp as a function of three parameters (start value, increment, offset) rather than storing the actual timestamp values. This makes compression applicable to time stamp data despite its unique values, as the parameters can be efficiently stored and retrieved
Solution Approach 2:
The patent creates a universal compression approach that works for both measurement columns and time stamp columns. The same parameter-based encoding technique that compresses measurement data can also compress timestamp data, making the compression methodology versatile across different data types in the time series database
3Quantity of substance
If large data sizes are used for time series data sets, then all measurements can be stored, but main system memory usage becomes extremely high limiting or prohibiting certain in-memory operations
Solution Approach 1:
The patent transforms the storage representation from full timestamp values to compact parameter sets, reducing memory footprint from 4-6 bytes per timestamp to 3 parameters per group. This enables time series data to be stored in main memory at feasible sizes, making in-memory operations practical and efficient
Solution Approach 2:
By segmenting time series data into groups with common encoding parameters, the patent reduces overall memory requirements. Each segment can be independently managed and processed, enabling efficient in-memory operations on segmented data rather than requiring the entire dataset to fit in memory
Data Source
AI summary
Time stamps for time series data can be efficiently compressed by grouping rows in a database table such that time stamp values of the rows in the group are ordered and characterizable by an increment and an offset, which can be stored for the set of rows respectively in an increment column and an offset column such that the time stamp values of the set of rows are represented by a single slope and offset. A run-length compression can be applied to the increment column and offset column for the table.