Parallel Hash Table Build Using Bitmap and Compact Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for building hash tables in non-partitioned hash joins are inefficient due to slow build processes and high memory allocation requirements, especially when the smaller join relation does not fit entirely into memory.
Innovation Solution
A parallel build process for compact, non-partitioned hash tables is implemented using multiple threads to scan and partition hash table keys and values, with cumulative counts determining space allocation and a bitmap structure to minimize collisions and memory usage, allowing for efficient construction of compact hash tables that can handle both partitioned and non-partitioned lookups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional partitioned join methods are used, then lookup efficiency is improved for large hash tables, but build time and memory overhead increase due to partitioning costs
Solution Approach 1:
The patent applies segmentation by dividing the hash table build process into independent thread-safe segments. Multiple threads can simultaneously build different partitions of the hash table without requiring global latches, as each thread maintains its own local state and uses atomic operations only for final merging. This resolves the contradiction by enabling parallel builds (reducing build time) while maintaining the partitioned structure needed for efficient lookups.
Solution Approach 2:
The patent uses preliminary action by pre-allocating memory space for each thread's partition before the build process begins. Each thread receives its designated portion of the hash table memory in advance, allowing independent parallel construction without dynamic memory allocation during the build phase. This eliminates memory overhead from repeated allocations while maintaining partitioned structure for efficient lookups.
2Productivity
If non-partitioned hash tables are used for small datasets, then build process is simpler and faster, but memory consumption increases and cache efficiency decreases
Solution Approach 1:
The patent applies dynamics by making the hash table structure adaptive to dataset size. The system automatically determines whether to use partitioned or non-partitioned structure based on the size of the input data, transitioning between modes dynamically. For small datasets, it uses compact non-partitioned tables with simple builds; for large datasets, it automatically partitions to reduce memory consumption and improve cache efficiency, while maintaining parallel build capability throughout.
3Loss of time
If parallel build processes are implemented, then build time is reduced, but synchronization overhead and complexity increase
Solution Approach 1:
The patent extracts the synchronization problem from the parallel build process by removing the need for global latches and complex inter-thread communication. Each thread operates independently with its own local hash table partition, eliminating the need for continuous synchronization during the build phase. Only a simple atomic merge operation is needed at the end, dramatically reducing synchronization overhead and complexity while maintaining parallel build speedup.
4Adaptability or versatility
If hash tables are built to handle n-m joins and left-outer joins, then join operation versatility is improved, but memory allocation requirements increase
Solution Approach 1:
The patent achieves universality by creating a single parallel hash table build mechanism that handles multiple join types (n-m joins, left-outer joins, and standard equijoins) through the same infrastructure. The partitioned parallel build structure naturally supports variable-degree joins by allowing threads to insert multiple keys mapping to the same value, and supports outer joins by maintaining insertion order information. This eliminates the need for separate memory allocations for different join types, as the same partitioned structure serves all join operations.
Data Source
AI summary
A method for creating a compact hash table includes a thread executing using a processor for scanning a subset of keys and values, and for each key, performing a hash operation and inserting hashed keys into a bitmap structure. Cumulative population counts of keys and values are determined within the bitmap. The method repeats scanning the subset of the keys and values. The keys and values are inserted into a compacted array using the cumulative population counts.