Columnar Database Compression Using Sorted Dictionaries and Dynamic Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database compression techniques, especially in cloud-based columnar databases, are computationally expensive and inefficient due to the need for delta merge operations and lack of optimal global compression optimization, leading to suboptimal storage efficiency and high computational costs.
Innovation Solution
A compression engine that generates dictionaries of unique values in sorted order for each data column, dynamically selects the most efficient compression technique (such as dictionary, null-value, single-value, prefix/suffix, run-length encoding, or frame of reference compression) based on dictionary characteristics, and allocates variable-sized persistent pages to store compressed data chunks, optimizing storage and computation by leveraging CPU caches and minimizing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression techniques are used in cloud-based columnar databases, then storage efficiency is improved, but computational cost increases due to delta merge operations
Solution Approach 1:
The patent applies preliminary action by performing compression operations during the initial data loading phase rather than requiring subsequent delta merge operations. Data is compressed into variable-sized persistent pages before being stored in the columnar database, eliminating the need for costly post-processing merge operations that traditionally required re-compression of merged data segments.
Solution Approach 2:
The patent segments data into variable-sized persistent pages that are allocated based on the actual compressed data size rather than fixed page sizes. This segmentation allows each data chunk to be compressed independently and stored in appropriately-sized pages, avoiding the computational overhead of merging and re-compressing data segments.
2Ease of manufacture
If fixed-size pages are used in columnar databases, then memory allocation is simplified, but storage efficiency decreases due to wasted space
Solution Approach 1:
The patent implements dynamic page sizing where persistent pages are allocated with sizes that correspond to the actual compressed data chunk sizes rather than using fixed page sizes. The system dynamically determines the appropriate page size based on the compression ratio and data characteristics, optimizing storage efficiency while maintaining simple allocation through direct size-based page selection.
3Quantity of substance
If multiple compression techniques are evaluated for each data column, then compression ratio is improved, but processing time increases
Solution Approach 1:
The patent changes parameters by evaluating multiple compression techniques (dictionary compression, run-length encoding, prefix/suffix compression, frame of reference compression) and selecting the one that produces the smallest compressed representation for each data column. The system adjusts compression parameters such as dictionary size thresholds and run-length thresholds to optimize the balance between compression ratio and processing time.
Data Source
AI summary
A method for compressing columnar data may include generating, for a data column included in a data chunk, a dictionary enumerating, in a sorted order, a first set of unique values included in the first data column. A compression technique for generated a compressed representation of the data column having a fewest quantity of bytes may be identified based at least on the dictionary. The compression technique including a dictionary compression applying the dictionary and/or another compression technique. A compressed data chunk may be generated by applying the compression technique to compress the data column included in the data chunk. The compressed data chunk may be stored at a database in a variable-size persistent page whose size is allocated based on the size of the compressed representation of the data column. Related systems and articles of manufacture are also provided.