Parallel Hash Join Lock-Free Index Map
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel computing environments with shared memory face inefficiencies in join operations due to sequential steps and synchronization requirements, which limit the effective exploitation of parallelism and processing power.
Innovation Solution
A lock-free, cache-efficient index hash map data structure is developed to facilitate parallel join operations by generating key-index pairs and utilizing a two-step process of local hash table creation and merging, allowing concurrent execution without locks and maximizing processor utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sequential operations and synchronization operations (locking) are used in parallel join processes, then data coherency is maintained, but processing speed and parallelism exploitation are reduced
Solution Approach 1:
The patent divides the join operation into distinct phases: hash table construction phase and join execution phase. During construction, multiple threads independently build hash tables without synchronization. During execution, threads independently probe and join data. This segmentation eliminates the need for locks while maintaining data coherency through phase separation.
Solution Approach 2:
The patent performs preliminary hashing and organization of data into hash tables before the actual join operation. This preliminary action prepares data in a structured format that allows subsequent parallel join operations to proceed without synchronization, as all threads operate on pre-organized data structures.
2Reliability
If locks are used to ensure data coherency in shared memory, then consistency is maintained, but processor utilization and parallelism are limited
Solution Approach 1:
The patent segments the parallel join process into independent construction and execution phases, allowing multiple processors to work simultaneously without requiring locks. Each processor independently contributes to hash table construction and then independently performs join operations, maximizing processor utilization while maintaining consistency through phase separation.
3Reliability
If synchronization operations are implemented in parallel join processes, then data coherency is ensured, but processing throughput is reduced
Solution Approach 1:
The patent divides the join operation into construction and execution phases, eliminating the need for synchronization operations. Multiple threads can simultaneously construct hash tables and simultaneously perform join operations on different data partitions, achieving high throughput without compromising data coherency.
Solution Approach 2:
The patent enables continuous parallel processing by eliminating synchronization barriers. Threads continuously perform useful work (hashing, probing, joining) without being blocked by lock operations, maintaining high throughput while ensuring data coherency through the structured two-phase approach.
Data Source
Figure 1
Figure 2
Figure 3A~3C
AI summary
According to some embodiments, a system and method for a parallel join of relational data tables may be provided by calculating, by a plurality of concurrently executing execution threads, hash values for join columns of a first input table and a second input table; storing the calculated hash values in a set of disjoint thread-local hash maps for each of the first input table and the second input table; merging the set of thread-local hash maps of the first input table, by a second plurality of execution threads operating concurrently, to produce a set of merged hash maps; comparing each entry of the merged hash maps to each entry of the set of thread-local hash maps for the second input table to determine whether there is a match, according to a join type; and generating an output table including matches as determined by the comparing.