Dynamic Compression for Hash Join Cache Fit
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hash joins in data processing often fail to efficiently fit into the cache hierarchy, leading to performance issues due to random accesses and memory constraints, particularly when the inner join data structure does not fit well within memory.
Innovation Solution
A method is introduced to dynamically compress join attributes using an encoding scheme that selects the optimal compression based on the data type of the join attributes, allowing for efficient representation and fitting into the cache hierarchy by encoding join keys in a compact form, such as logarithmic bits for numerical values and suffix-based encoding for strings, and using bitmaps for filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hash joins are performed with traditional data structures, then join operations can be executed, but the data structures do not fit well into the cache hierarchy, leading to performance degradation
Solution Approach 1:
The patent applies parameter changes by dynamically selecting compression schemes based on data types and characteristics. Different compression parameters (encoding methods, bit widths) are adjusted to optimize the fit of hash table data into cache memory, thereby improving cache hit rates and reducing access time without sacrificing join operation correctness
Solution Approach 2:
The patent implements dynamics through dynamic compression selection during hash join execution. The system adapts compression strategies based on runtime conditions such as data type, data distribution, and available memory, allowing the hash table structure to dynamically adjust its compression level to maximize cache utilization during join operations
2Volume of stationary object
If data is compressed to fit into cache, then cache hierarchy fit is improved, but additional compression overhead and complexity are introduced
Solution Approach 1:
The patent applies local quality by selecting different compression schemes for different data types and characteristics within the hash table. Instead of uniformly compressing all data, the system applies compression locally based on the specific properties of each column or data segment, optimizing space utilization while maintaining manageable complexity through targeted rather than universal compression strategies
Solution Approach 2:
The patent implements partial action by applying compression selectively to only those portions of the hash table data that benefit most from compression. The system identifies and compresses specific columns or data segments where compression provides the greatest space savings, rather than compressing the entire data structure, thereby reducing the overall complexity burden
3Quantity of substance
If compression is applied to join attributes, then memory efficiency is improved, but encoding and decoding overhead increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing compressed representations of join attributes during the hash table build phase. The compression is performed in advance before the actual join operation, so that during the probe phase, the system only needs to perform efficient lookups on already-compressed data, minimizing the time overhead during critical join execution
Solution Approach 2:
The patent implements copying by creating compressed copies of the original join attributes for storage in the hash table. Instead of storing and manipulating the full original data, the system works with compressed copies that occupy less memory, while maintaining the ability to reconstruct or access the necessary information for join operations
Data Source
AI summary
A method includes joining data between at least two data sets. Values of one or more join attributes of each of the data sets is represented in a compressed form, indicated by an encoding scheme. A compression scheme for the one or more join attributes is dynamically selected.