Parallel Hash Join Optimization via Multi-Core Task Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face performance issues with hash joins due to unfavorable memory access patterns and slow execution times, especially when processing large data sets in parallel processing computer systems.
Innovation Solution
The method optimizes hash join operations by distributing join tasks across multiple CPU cores using multi-threaded design and record aggregation techniques, leveraging parallel processing capabilities without requiring substantial software redesign, and employs a hash join technique that partitions data into hash buckets for asynchronous processing across multiple cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hash join algorithm is used to process large data sets, then join operation can be performed with expected complexity O(M + N), but memory access patterns become unfavorable and execution speed decreases
Solution Approach 1:
The patent divides the hash join operation into multiple segments by creating separate hash tables for different portions of the data. Each hash table handles a specific segment of the input data, allowing parallel processing while maintaining the overall join operation efficiency. This segmentation resolves the contradiction by enabling multiple threads to work simultaneously on different data segments without contending for the same memory resources.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing hash tables in a multi-dimensional structure that allows simultaneous access from multiple processing threads. Instead of a single sequential hash table, the system creates a distributed hash table structure where data can be accessed along multiple dimensional axes, thereby improving execution speed while maintaining join operation complexity.
2Productivity
If traditional hash join is implemented in parallel processing systems, then data processing capacity increases, but synchronization overhead and complexity increase
Solution Approach 1:
The patent extracts the synchronization requirement from the hash join operation by creating independent hash tables that can be built and queried simultaneously without mutual interference. Each processing thread works with its own dedicated hash table structure, eliminating the need for complex synchronization mechanisms while maintaining parallel processing capacity.
Solution Approach 2:
The patent introduces an intermediary data structure that acts as a buffer between parallel processing threads and the final join results. This intermediary structure allows multiple threads to contribute their processed data without direct interaction, reducing synchronization overhead while maintaining data integrity and processing capacity.
3Quantity of substance
If hash join processes large volumes of data from multiple sources, then comprehensive data analysis is achieved, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-building hash tables from the input data before the actual join operation begins. This preliminary hashing and organization of data allows the subsequent join operation to proceed rapidly by simply comparing pre-organized structures, rather than performing complex comparisons on raw data, thereby reducing processing time for large data volumes.
Solution Approach 2:
The patent maintains continuity of useful action by ensuring that all processing threads are constantly engaged in productive work throughout the join operation. The parallel hash table structure allows continuous data flow and processing without idle waiting or sequential bottlenecks, maximizing the utilization of processing resources and reducing overall processing time for large datasets.
Data Source
Figure 1
Figure 2A~2B
Figure 3A
AI summary
Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings in parallel to determine whether to join the records in the groupings. A thread joins two data records under consideration if the hashes associated with the data records match. The joined data records are output.