Columnar Data Compression with Variable-Size Persistent Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database compression techniques, especially in cloud-based columnar databases, are computationally expensive and inefficient due to the need for delta merge operations and lack of global compression optimization, leading to suboptimal storage efficiency and high computational costs.
Innovation Solution
A compression engine that generates dictionaries of unique values in sorted order to select the most efficient compression techniques for each data chunk, such as dictionary compression, null-value compression, single-value compression, prefix/suffix compression, run-length encoding, or frame of reference compression, based on the characteristics of the data, and allocates persistent pages accordingly to minimize computational resources and maximize storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional compression techniques are used in cloud-based columnar databases, then storage efficiency is improved, but computational costs increase significantly due to delta merge operations
Solution Approach 1:
The patent segments the data storage into immutable data chunks with fixed compression techniques, eliminating the need for expensive delta merge operations. Each chunk is independently compressed and stored, allowing parallel processing and reducing computational overhead while maintaining storage efficiency.
Solution Approach 2:
The patent applies compression techniques preliminarily during data ingestion before data is stored in immutable chunks. By pre-compressing data with appropriate techniques (dictionary compression, run-length encoding, etc.) based on data characteristics, the system avoids subsequent computational expenses of re-compression during merge operations.
2Quantity of substance
If global compression optimization is implemented, then compression ratios are improved, but device complexity increases due to coordination overhead
Solution Approach 1:
The patent implements local compression optimization by analyzing data characteristics within each immutable chunk and applying the most suitable compression technique locally. Each chunk can use different compression methods (dictionary compression for text, run-length encoding for repetitive data, etc.) based on its specific data patterns, achieving high compression ratios without global coordination complexity.
Solution Approach 2:
The system performs self-service compression optimization by automatically analyzing data characteristics and selecting appropriate compression techniques without external coordination. The compression engine autonomously evaluates data patterns and applies optimal compression methods, eliminating the need for complex global optimization coordination while maintaining high compression ratios.
3Quantity of substance
If multiple compression techniques are evaluated for each data chunk, then compression efficiency is improved, but processing time increases
Solution Approach 1:
The patent evaluates multiple compression techniques partially by first applying quick assessment metrics to data chunks and only performing full compression evaluations for techniques that show promise based on initial data characteristics. This selective evaluation approach maintains high compression efficiency while significantly reducing processing time compared to evaluating all techniques exhaustively.
Data Source
AI summary
A method for compressing columnar data may include generating, for a data column included in a data chunk, a dictionary enumerating, in a sorted order, a first set of unique values included in the first data column. A compression technique for generated a compressed representation of the data column having a fewest quantity of bytes may be identified based at least on the dictionary. The compression technique including a dictionary compression applying the dictionary and/or another compression technique. A compressed data chunk may be generated by applying the compression technique to compress the data column included in the data chunk. The compressed data chunk may be stored at a database in a variable-size persistent page whose size is allocated based on the size of the compressed representation of the data column. Related systems and articles of manufacture are also provided.