Dynamic Redistribution Matrix for Parallel Join Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In massively parallel processing systems, conventional methods for parallel joins in relational database management systems result in high processing costs due to static redistribution of rows based on pre-configured hash value-to-processing module mapping, which negatively impacts overall processing efficiency.
Innovation Solution
A system and method that dynamically calculates and optimizes the redistribution matrix by determining row counts for each processing node based on join attributes, generating an optimized redistribution matrix to minimize the number of rows redistributed during a join operation, using techniques like the Hungarian algorithm to maximize the diagonal sum of the redistribution matrix.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If rows are statically redistributed based on pre-configured hash value-to-processing module mapping, then the join operation can be executed in parallel across multiple processing modules, but the processing cost becomes high and processing efficiency deteriorates
Solution Approach 1:
The patent transforms the static redistribution approach into a dynamic one by calculating an optimized redistribution matrix based on actual row counts and join attributes. The system determines the number of rows to redistribute from each processing module to every other processing module, creating a dynamic optimization layer between the static hash mapping and the actual data redistribution, thereby reducing unnecessary data movement and processing costs
Solution Approach 2:
The patent changes the parameter of redistribution strategy from fixed hash-based mapping to variable optimized matrix-based mapping. By computing the redistribution matrix using row counts and join attributes, the system adapts the redistribution parameters to the specific characteristics of each query, allowing flexible optimization of data movement to minimize processing costs while maintaining parallel execution
2Ease of operation
If rows are redistributed to processing modules based solely on hash value, then the redistribution process is simple and fast, but the number of rows redistributed is excessive and communication cost increases
Solution Approach 1:
The patent applies preliminary action by pre-calculating the optimized redistribution matrix before executing the join operation. The system computes the matrix based on row counts and join attributes in advance, determining the optimal target processing module for each row before actual redistribution occurs. This preliminary optimization prevents excessive data movement during the join execution while maintaining operational simplicity through automated calculation
3Speed
If conventional hash redistribution is used, then rows can be distributed to processing modules quickly, but query response time increases due to high processing cost
Solution Approach 1:
The patent introduces a dynamic optimization layer that calculates the redistributed row counts and generates an optimized redistribution matrix before executing the join. This dynamic approach adapts to the specific query characteristics and data distribution, minimizing the actual number of rows that need to be communicated between processing modules, thereby reducing query response time while preserving fast data distribution capabilities
Data Source
AI summary
A system, method, and computer-readable medium for optimizing execution of a join operation in a parallel processing system are provided. A plurality of processing nodes that have at least one row of one or more tables involved in a join operation are identified. For each of the processing nodes, respective counts of rows that would be redistributed to each of the processing nodes based on join attributes of the rows are determined. A redistribution matrix is calculated from the counts of rows of each of the processing nodes. An optimized redistribution matrix is generated from the redistribution matrix, wherein the optimized redistribution matrix provides a minimization of rows to be redistributed among the nodes to execute the join operation.