Small Table Replication for Parallel Join Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In relational database systems, the broadcast distribution method for parallel join operations incurs high data communication protocol overhead, especially when a small left-side table needs to be broadcast to a large number of computational units, leading to scalability bottlenecks and increased resource consumption.
Innovation Solution
The method involves replicating the smaller left-side table and retrieving both the small table and a portion of the larger right-side table, thereby incurring only one unit of protocol overhead, reducing the need for multiple producer computational units and optimizing data communication by encapsulating the request within the request for the right-side table portion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the small left-side table is broadcast to all computational units, then the join operation can be performed in parallel across all units, but the data communication protocol overhead increases significantly
Solution Approach 1:
The patent segments the right-side table into multiple partitions distributed across different computational units. Instead of broadcasting the small left-side table to all units, each unit processes only its local partition of the right-side table, eliminating the need for widespread broadcasting and reducing protocol overhead while maintaining parallel processing capability.
Solution Approach 2:
The patent applies local quality by having each computational unit process only the relevant portion of data local to it. Each unit performs the join operation between the small left-side table and its local partition of the right-side table, avoiding unnecessary communication with other units and reducing overall protocol overhead.
2Productivity
If the small left-side table is broadcast to a large number of computational units, then the join operation achieves high parallelism, but the resource consumption increases
Solution Approach 1:
The patent segments the computational work by dividing the right-side table into multiple partitions distributed across different computational units. This allows parallel processing to be achieved without requiring the small left-side table to be replicated and transmitted to all units, thereby reducing resource consumption while maintaining high degree of parallelism.
3Ease of operation
If the small left-side table is broadcast to all computational units, then the join operation can proceed without distribution overhead for the left table, but the interconnect protocol overhead becomes a scalability bottleneck
Solution Approach 1:
The patent segments the data distribution strategy by keeping the small left-side table localized and dividing the right-side table into partitions. This eliminates the need for complex broadcasting operations across the interconnect, reducing protocol overhead and improving scalability while maintaining operational simplicity through a clear partition-based execution model.
Data Source
AI summary
A method, system, and computer program product for planning relational database joins in systems with multiple parallel computational units. The computer implemented method compiles an execution plan for dissemination to a plurality of execution units where the plan combines a smaller left-side table with a portion of a larger right-side object to reduce system overhead. Upon receiving a request for a join operation comprising left-side table data and right-side table data where the left-side table data is much smaller than the right-side table data, then the method compiles a plan to apportion an entirety of the left-side table data and only a portion of the right-side object to a plurality of computational units. The method continues by sending distribution instructions to respective computational units where the distribution instructions include retrieval of the entirety of the left-side table data with the retrieval of only a portion of the right-side object.