Order-Preserving String Dictionary Compression for Column Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Column-oriented database systems face inefficiencies when dealing with large or unknown domain sizes for variable-length string attributes, as existing compression schemes either generate expensive variable-length codes or difficult-to-extend fixed-length codes, particularly when the domain size changes over time.
Innovation Solution
The implementation of a data structure that supports order-preserving dictionary compression using a shared-leaves structure, where variable-length string values are mapped to fixed-length integer codes, enabling efficient encoding and decoding, and allowing for bulk loading and query processing with optimized indexing strategies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If variable-length codes are used for compression, then compression ratio is improved, but code complexity and processing overhead increase
Solution Approach 1:
The patent segments the domain into multiple pages, each page containing a subset of the dictionary. This allows variable-length codes to be used within each page while keeping the code structure manageable and enabling efficient processing through page-level operations rather than handling the entire domain at once.
Solution Approach 2:
The patent introduces a page dimension to organize the dictionary space, transforming a one-dimensional flat structure into a two-dimensional hierarchical structure (pages × entries per page). This enables efficient compression by allowing variable-length codes within pages while using fixed-length page identifiers, thereby reducing overall code complexity.
2Speed
If fixed-length codes are used for compression, then processing speed is improved, but adaptability to changing domain sizes deteriorates
Solution Approach 1:
The patent implements dynamic code length selection based on the domain size within each page. When the domain size is small, variable-length codes provide better compression; when the domain size is large or changes over time, the system can switch to fixed-length codes or adjust the page granularity, thereby maintaining both processing speed and adaptability.
Solution Approach 2:
The patent changes the code length parameter dynamically based on the domain characteristics. By adjusting the code length within pages according to the actual domain size and change patterns, the system achieves both fast processing (through optimized code lengths) and adaptability (through parameter adjustment).
3Adaptability or versatility
If dictionary size increases to handle large domain, then coverage is improved, but memory usage increases
Solution Approach 1:
The patent segments the large dictionary into multiple smaller pages, each fitting within available memory constraints. This segmentation allows the system to handle large domains by loading and processing pages incrementally, thereby improving domain coverage without requiring the entire dictionary to reside in memory simultaneously.
Solution Approach 2:
The patent applies different storage and processing qualities to different parts of the dictionary based on local characteristics. Frequently accessed pages or pages with smaller domains can be kept in memory with higher quality representations, while less frequently accessed pages can be stored externally or compressed more aggressively, thereby optimizing memory usage while maintaining overall domain coverage.
4Speed
If order-preserving compression is implemented, then query performance is improved, but data structure complexity increases
Solution Approach 1:
The patent maintains order-preserving properties within each page rather than across the entire dictionary. This segmentation reduces the complexity of maintaining global order while still enabling efficient query performance through local ordering within pages, allowing range queries and sorting operations to be performed efficiently on compressed data.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and systems are described that involve usage of dictionaries for compressing a large set of variable-length string values with fixed-length integer keys in column stores. The dictionary supports updates (e.g., inserts of new string values) without changing codes for existing values. Furthermore, a shared-leaves approach is described for indexing such a dictionary that compresses the dictionary itself while offering access paths for encoding and decoding.