Type-Specific Database Compression for Time-Series Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face challenges in efficiently storing and processing high-volume, high-write-rate time-series data, particularly in scenarios like IoT and industrial applications, where they struggle with scalability and query efficiency, and existing compression methods either increase storage costs or compromise performance.
Innovation Solution
A database system that implements a plurality of compression policies to convert data from a row-based representation to a columnar form, using various compression schemes like LZ, DEFLATE, and delta encoding, and stores compressed data in mini-sets, allowing for efficient storage and query processing while maintaining compatibility with traditional row-based systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in row-based format in conventional database systems, then query processing is straightforward, but storage capacity requirements increase and compression efficiency deteriorates
Solution Approach 1:
The patent segments data into columnar mini-sets within row-based tables, allowing selective compression of individual columns while maintaining row-based query capabilities. This segmentation enables efficient storage of time-series data by grouping similar data types together for compression without requiring a complete architectural shift to columnar storage.
Solution Approach 2:
The patent changes the compression parameters by applying different compression policies to different columns based on their data characteristics. Time-series columns use specialized compression algorithms while other columns use standard compression, optimizing storage efficiency without compromising query performance on non-compressed columns.
2Quantity of substance
If compression is applied to reduce storage costs, then storage capacity needs decrease, but system complexity increases
Solution Approach 1:
The database system automatically selects and applies appropriate compression policies to columns based on their data types and characteristics. This self-service approach eliminates the need for manual compression policy configuration, reducing system complexity while achieving optimal compression ratios for different data types.
Solution Approach 2:
The patent applies different compression policies to different columns based on their local characteristics. Time-series columns with high cardinality use one compression approach, while columns with low cardinality use another, optimizing compression efficiency without requiring a single complex global policy.
3Quantity of substance
If columnar storage is used to improve compression ratios, then storage efficiency increases, but compatibility with traditional row-based systems decreases
Solution Approach 1:
The patent merges columnar compression techniques with row-based storage architecture by implementing columnar mini-sets within row-based tables. This hybrid approach maintains compatibility with traditional row-based database systems while incorporating the storage efficiency benefits of columnar organization for specific data types.
Solution Approach 2:
The patent creates a multi-functional storage system that supports both row-based and columnar access patterns within the same table structure. The columnar mini-sets can be queried efficiently for analytics while the overall table structure maintains row-based compatibility, serving multiple access patterns simultaneously.
Data Source
AI summary
A database system stores both compressed and uncompressed data in a row-based database system. The database system converts a representation of data involving a set of rows of a source database table, each row comprising multiple values, into a representation involving a single row stored in a target database table, each column of the row comprising arrays of values from the set. The database system may perform type-specific compression of data when storing in the target database table. Accordingly, the database system may apply different compression schemes for different columns or sets of values obtained from the source database table and may group or order selected rows or store additional summary information to improve query performance to the target database table. The database system allows users to query data stored in compressed form. The database system combines compressed and uncompressed data at query time for efficient database analytics.