Type-Specific Compression for Time-Series Query and Storage Tradeoffs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face challenges in supporting high write rates and efficient querying of time-series data, particularly in scenarios with high ingest rates and complex query patterns, while also managing storage costs effectively.
Innovation Solution
A database system that implements a plurality of compression policies to store and process time-series data by converting rows into a compressed array form, allowing for efficient storage and querying, and utilizes a hypertable partitioned across servers and storage devices to scale and handle queries efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in uncompressed form in a row-based database system, then query processing is simpler and faster, but storage capacity requirements increase significantly
Solution Approach 1:
The patent segments data storage by creating separate columnar storage structures for each attribute within rows. This allows the database to store compressed versions of individual columns while maintaining the ability to query any column independently, thus reducing overall storage requirements without significantly impacting query performance.
Solution Approach 2:
The patent applies compression policies that change the physical representation of data parameters. By transforming data from its original uncompressed format to compressed formats (such as run-length encoding, delta encoding, or other compression algorithms), the system reduces storage capacity requirements while implementing policies to manage the trade-off between compression ratio and query performance.
2Quantity of substance
If column-store compression techniques are used, then storage capacity is reduced, but insert performance deteriorates due to additional processing overhead
Solution Approach 1:
The patent performs preliminary compression actions during the data insertion process itself, rather than as a separate post-processing step. By applying compression policies at the time of data insertion and maintaining compressed storage structures, the system reduces the performance penalty typically associated with compression. The database engine is designed to handle compression as part of the normal insert operation, thereby maintaining higher insert rates compared to systems that compress data after insertion.
3Productivity
If data is stored in row-based format, then insert operations are simpler and faster, but storage efficiency decreases for time-series data with repeating patterns
Solution Approach 1:
The patent segments the row-based data structure into columnar components, allowing each column to be stored and compressed independently. This segmentation enables the database to maintain the simplicity of row-based insertion operations while applying compression techniques to specific columns that exhibit repeating patterns, thereby improving storage efficiency without significantly complicating the insert process.
Solution Approach 2:
The patent applies compression selectively to specific columns or attributes based on their characteristics. By analyzing which columns contain time-series data with repeating patterns and applying compression policies only to those columns, the system improves storage efficiency locally where needed while maintaining the overall simplicity of row-based data insertion and processing.
Data Source
AI summary
A database system stores both compressed and uncompressed data in a row-based database system. The database system converts a representation of data involving a set of rows of a source database table, each row comprising multiple values, into a representation involving a single row stored in a target database table, each column of the row comprising arrays of values from the set. The database system may perform type-specific compression of data when storing in the target database table. Accordingly, the database system may apply different compression schemes for different columns or sets of values obtained from the source database table and may group or order selected rows or store additional summary information to improve query performance to the target database table. The database system allows users to query data stored in compressed form. The database system combines compressed and uncompressed data at query time for efficient database analytics.