Database Parallel Execution Clumping for Communication Cost Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face inefficiencies in parallel execution of user-defined table functions due to unnecessary data redistribution and complex query execution plans, which increase computational and communication costs.
Innovation Solution
The solution involves determining compatibility of data partitioning methods between operations and user-defined table functions, allowing for 'clumping' of these operations into the same execution sub-plan, thereby avoiding unnecessary data redistribution and reducing communication costs by executing them on the same set of processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data redistribution is performed between operations and user-defined table functions, then data can be properly partitioned for parallel execution, but computational and communication costs increase
Solution Approach 1:
The patent merges operations with compatible data partitioning requirements into the same execution sub-plan, eliminating the need for data redistribution between them. This is achieved by detecting compatibility between partitioning methods and consolidating operations that can execute together on the same processes, thereby reducing communication overhead while maintaining correct data partitioning.
2Adaptability or versatility
If multiple operations are executed separately in parallel, then flexibility in execution is maintained, but query execution plan complexity increases
Solution Approach 1:
The patent combines multiple operations into unified execution sub-plans when their data partitioning requirements are compatible. This merging reduces the number of separate execution units and simplifies the overall query execution plan structure while preserving the ability to handle diverse partitioning strategies through the compatibility detection mechanism.
3Reliability
If data redistribution table queues are used for all operations, then data partitioning is ensured, but computational costs increase
Solution Approach 1:
The patent extracts and eliminates unnecessary data redistribution table queues from the execution plan. By detecting when operations have compatible partitioning requirements, the system removes the intermediate redistribution step entirely, allowing operations to execute directly on the same processes without the computational overhead of data redistribution.
Data Source
AI summary
While generating an execution plan for a database statement that includes a user-defined table function that requires partitioning of input data to the function, it is determined whether the partitioning of output data from an operation that feeds the function is compatible with the partitioning required by the function. If the partitioning of output data from the operation that feeds the function is compatible with the partitioning required by the function, then the execution plan is constructed to specify execution of both (1) the function and (2) the operation that feeds the function, by a same set of processes. For example, the operation that feeds the function and the function itself are clumped into the same sub-plan or sub-tree, which is scheduled and run in parallel on a set of parallel execution slaves.