Split Dictionary Compression for Faster Query Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression techniques for dictionary tables face challenges such as increased complexity in decompression, performance degradation during queries, and inefficient use of storage space due to variable length columns, especially as dictionary sizes grow, leading to difficulties in maintaining high compression ratios and efficient data access.
Innovation Solution
The implementation of a split dictionary approach, where dictionary tables are divided into major and minor dictionaries based on string length distributions, using block encoding vectors to optimize search performance and reduce memory usage, along with modified delta encoding methods to maintain efficient data access and compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression algorithms (Huffman, null suppression) are used to achieve high compression ratios, then storage space is reduced, but decompression complexity and processing time increase significantly
Solution Approach 1:
The patent divides the dictionary into multiple segments (frequent items, infrequent items, and a hybrid section) and applies different compression techniques to each segment. This segmentation allows the system to achieve high compression ratios for frequent items while maintaining simple decompression for the hybrid section, thus resolving the contradiction between compression ratio and decompression complexity.
Solution Approach 2:
Different compression strategies are applied to different parts of the dictionary based on their access patterns and characteristics. The frequent items section uses aggressive compression, the hybrid section uses a balanced approach with direct indexing, and the infrequent items section uses simpler compression. This local quality approach optimizes both compression ratio and decompression complexity for each specific region.
2Quantity of substance
If high compression ratios are achieved using conventional techniques, then storage efficiency improves, but query performance and data access speed deteriorate
Solution Approach 1:
The patent pre-processes the dictionary data during initialization to create the hybrid section with direct indexing information and to establish the segmented structure. This preliminary action ensures that frequently accessed items are readily available during queries without requiring full decompression, thus improving query performance while maintaining high storage efficiency.
Solution Approach 2:
The hybrid section acts as an intermediary between the heavily compressed frequent items section and the infrequent items section. It contains a subset of frequently accessed items in a partially decompressed or directly indexable format, serving as a buffer that improves query performance without sacrificing overall storage efficiency.
3Quantity of substance
If dictionary size increases to accommodate growing enterprise data, then data coverage and analytics capability improve, but memory consumption and processing overhead increase
Solution Approach 1:
The patent implements a dynamic dictionary structure where the hybrid section can be adjusted based on access patterns and available memory resources. The system can dynamically determine which items move between sections and adjust the sizing of each segment, allowing the dictionary to adapt to growing data while optimizing memory consumption for the current workload.
4Adaptability or versatility
If variable length columns are used in dictionary tables to accommodate different data, then data flexibility and representational accuracy improve, but storage space utilization and compression efficiency worsen
Solution Approach 1:
The patent segments the dictionary into fixed-width and variable-width sections. The hybrid section uses fixed-width encoding for commonly accessed items, ensuring efficient storage and fast access, while the infrequent items section can use variable-length encoding where needed. This segmentation allows the system to maintain data flexibility while optimizing storage space utilization for the majority of accesses.
Data Source
AI summary
Methods, apparatus, systems, and computer program products for utilizing a split dictionary are described. A mean and a standard deviation of a length of a plurality of records in a dictionary may be computed. A split condition may be determined based on the computed mean and the computed standard deviation. The dictionary may be split into a plurality of split dictionaries based on the split condition.