Order-Preserving String Compression for Growable Column Dictionaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Column-oriented database systems face inefficiencies in handling large domain sizes for variable-length string attributes, as existing compression schemes either generate expensive variable-length codes or fixed-length codes that are difficult to extend, especially when the domain size is not known in advance.
Innovation Solution
The implementation of a data structure that supports order-preserving dictionary compression using a shared-leaves structure, where variable-length string values are encoded into fixed-length integer codes, allowing for efficient data loading, query compilation, and query execution, with the ability to handle changing domain sizes by propagating string values and generating new codes as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If variable-length codes are used for compression, then compression ratio is improved, but decoding complexity and processing time increase
Solution Approach 1:
The patent changes the parameter of code length from variable to fixed, where all codes are standardized to a uniform length (e.g., 4 bytes). This eliminates the need for variable-length decoding logic while maintaining compression effectiveness through dictionary-based encoding, thus reducing decoding complexity without sacrificing compression ratio
Solution Approach 2:
The patent segments the data handling process into two distinct phases: encoding phase where variable-length strings are mapped to fixed-length codes, and decoding phase where fixed-length codes are efficiently converted back. This segmentation allows optimization of each phase independently, with the fixed-length codes simplifying the decoding segment
2Speed
If fixed-length codes are used for compression, then decoding speed is improved, but adaptability to changing domain sizes deteriorates
Solution Approach 1:
The patent introduces dynamic adaptability into the fixed-length code system through a growable dictionary structure. When new string values are encountered during encoding, the dictionary automatically expands to accommodate them, generating new fixed-length codes as needed. This allows the system to adapt to changing domain sizes while maintaining the speed benefits of fixed-length decoding
Solution Approach 2:
The fixed-length code structure serves multiple functions: it provides fast decoding performance, enables efficient cache utilization, and simultaneously supports dynamic adaptation to growing domain sizes through the extensible dictionary. The same code structure handles both static and dynamic scenarios
3Adaptability or versatility
If dictionary size increases to handle large domain sizes, then coverage of string values is improved, but memory usage increases
Solution Approach 1:
The patent changes the parameter of code length to fixed-length (e.g., 4 bytes per code), which significantly reduces the memory footprint compared to storing variable-length strings directly in the dictionary. This allows the dictionary to grow larger to cover more string values while keeping the encoded representation compact and memory-efficient
Solution Approach 2:
The patent creates a compact copy of the string data in the form of fixed-length integer codes. Instead of storing the actual variable-length strings in the encoded dataset, only the compact integer codes are stored, with the full string values maintained only in the dictionary structure. This copying approach dramatically reduces memory usage for the encoded data
4Speed
If data is stored in sort order for efficient querying, then query performance is improved, but data insertion complexity increases
Solution Approach 1:
The patent performs preliminary sorting of string values during the dictionary construction phase, before the actual data encoding and storage. By pre-sorting the dictionary entries and assigning codes in sorted order, the system ensures that the encoded data naturally maintains sort order, eliminating the need for complex insertion logic during data loading while preserving query performance benefits
Data Source
AI summary
Methods and systems are described that involve usage of dictionaries for compressing a large set of variable-length string values with fixed-length integer keys in column stores. The dictionary supports updates (e.g., inserts of new string values) without changing codes for existing values. Furthermore, a shared-leaves approach is described for indexing such a dictionary that compresses the dictionary itself while offering access paths for encoding and decoding.