Multi-Table Theta Join Field Segmentation for Data Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In relational database systems with a distributed computing framework, multi-table theta join operations result in excessive data computation, leading to poor performance, high computing overheads, and increased memory and network transmission overheads due to the calculation of a Cartesian product for all fields involved in the query statement.
Innovation Solution
A control method that decomposes fields in first type tables into multiple field groups based on preset conditions, reducing the data amount calculated during the join operation, and generates an execution plan to read and process these groups efficiently, thereby reducing network transmission and memory overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a theta join operation is performed on all fields in tables used in the query statement, then a complete join result is obtained, but the data computation amount explodes due to Cartesian product calculation
Solution Approach 1:
The patent segments the fields of tables participating in theta join operations into multiple field groups based on preset conditions. Instead of performing a single Cartesian product on all fields, the system divides the computation into multiple smaller Cartesian products on field groups, then performs sequential join operations. This segmentation reduces the explosive data computation amount while maintaining complete join results through multi-step processing.
2Reliability
If a theta join operation is performed on all fields in tables used in the query statement, then a complete join result is obtained, but computing overheads are greatly increased
Solution Approach 1:
The patent segments the theta join operation into multiple smaller join operations on field groups. Each segment processes a subset of fields, reducing the computational complexity of each individual operation. The system generates an execution plan that processes these segments sequentially, lowering overall computing overheads while ensuring the complete join result is achieved through the combination of all segments.
3Reliability
If a theta join operation is performed on all fields in tables used in the query_statement, then a complete join result is obtained, but memory overheads are greatly increased
Solution Approach 1:
The patent segments the join computation into multiple field groups that are processed in sequence. Each segment loads only the necessary field data into memory at that moment, rather than loading all fields simultaneously. This segmentation approach significantly reduces peak memory overheads while maintaining the ability to produce complete join results through the sequential processing and combination of all field group segments.
4Reliability
If a theta join operation is performed on all fields in tables used in the query_statement, then a complete join result is obtained, but inter-node network transmission overheads are greatly increased
Solution Approach 1:
The patent segments the join operation into multiple field groups processed in sequence, where each segment generates a partial result that is transmitted between nodes. Instead of transmitting all intermediate data for a single large Cartesian product, the system transmits smaller partial results from each field group segment. This segmentation reduces the volume of inter-node network transmission overheads while ensuring complete join results are obtained through the aggregation of all transmitted segments.
Data Source
AI summary
A control method for performing a multi-table join operation and a corresponding apparatus are disclosed. Fields in a table whose fields participating in a theta join operation and used in the query statement meet a first preset condition are decomposed, to obtain a plurality of first field groups, so that the theta join operation can be implemented in steps in a form of the plurality of field groups. This can reduce a data amount of Cartesian product calculation during one join operation, greatly reduce network transmission overheads, computing overheads, and memory overheads, and improve execution efficiency.