Columnar Data Compression With Variable-Size Persistent Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database compression techniques, especially in cloud-based columnar databases, are computationally expensive and inefficient due to the need for delta merge operations and lack of optimal global compression optimization across entire database tables, leading to suboptimal storage efficiency and high computational costs.
Innovation Solution
A compression engine that processes data in chunks, generates dictionaries of unique values, and selectively applies compression techniques like dictionary compression, null-value compression, single-value compression, prefix/suffix compression, run-length encoding, and frame of reference compression based on dictionary characteristics to achieve the smallest compressed representation, allowing for variable-sized persistent pages and minimizing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression techniques are applied to columnar database tables, then storage efficiency is improved, but computational cost increases significantly due to delta merge operations and lack of global optimization
Solution Approach 1:
The patent segments the database table into multiple data chunks, each independently compressed using optimal techniques. This allows parallel processing of chunks without requiring expensive global delta merge operations, reducing computational cost while maintaining storage efficiency.
Solution Approach 2:
The system performs preliminary analysis of each data chunk to identify its characteristics (null density, value repetition patterns, data types) before applying compression. This preliminary characterization enables selection of the most appropriate compression technique for each chunk, avoiding unnecessary computational overhead from trying multiple techniques or performing global optimization passes.
2Quantity of substance
If multiple compression techniques are evaluated for each data column, then compression ratio is improved, but processing time increases
Solution Approach 1:
The patent applies different compression techniques to different data chunks based on their local characteristics rather than using a uniform approach across the entire table. Each chunk is analyzed for its specific properties (null density, value repetition, data types) and the most suitable compression technique is selected locally, achieving high compression ratios without the time cost of evaluating all techniques globally.
Solution Approach 2:
The system changes compression parameters and technique selection based on the characteristics of each data chunk. By analyzing chunk-specific parameters such as null density, value repetition patterns, and data types, the system dynamically selects and configures compression techniques to achieve optimal compression ratios while minimizing processing time through targeted rather than exhaustive evaluation.
3Ease of manufacture
If fixed-size pages are used in columnar databases, then memory management is simplified, but storage efficiency decreases due to wasted space in variable-sized compressed data
Solution Approach 1:
The patent transitions from static fixed-size pages to dynamic variable-sized pages that adapt to the actual size of compressed data chunks. Each page is allocated the exact size needed to hold its compressed content, eliminating wasted space while maintaining efficient memory management through the use of a page cache that handles variable-sized allocations.
4Productivity
If compression is applied at the table level rather than chunk level, then fewer compression operations are needed, but the ability to optimize for different data patterns is reduced
Solution Approach 1:
The patent segments the table into multiple data chunks that can be independently compressed and stored. This segmentation allows the system to apply different compression techniques to different chunks based on their characteristics, providing optimization flexibility while maintaining operational efficiency through parallel processing of segments.
Solution Approach 2:
The system creates a universal compression framework that can handle multiple data patterns and characteristics within the same table by applying different techniques to different chunks. This multi-functional approach allows the system to optimize for various data types, null densities, and repetition patterns simultaneously, achieving both efficiency and adaptability.
Data Source
AI summary
A method for compressing columnar data may include generating, for a data column included in a data chunk, a dictionary enumerating, in a sorted order, a first set of unique values included in the first data column. A compression technique for generated a compressed representation of the data column having a fewest quantity of bytes may be identified based at least on the dictionary. The compression technique including a dictionary compression applying the dictionary and/or another compression technique. A compressed data chunk may be generated by applying the compression technique to compress the data column included in the data chunk. The compressed data chunk may be stored at a database in a variable-size persistent page whose size is allocated based on the size of the compressed representation of the data column. Related systems and articles of manufacture are also provided.