Adaptive Granule Generation for Parallel Query Execution Skew
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Bloom partition pruning in Database Management Systems (DBMS) leads to execution skew and longer query execution times due to idle slave processes, as it prunes a significant portion of the workload, leaving available resources unused.
Innovation Solution
Adaptive granule generation and runtime data pruning, where the query coordinator regenerates work granules to ensure each available slave process is assigned work, distributing the workload more evenly among parallel processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If bloom partition pruning is used to filter out superfluous work granules, then query execution efficiency is improved by avoiding unnecessary processing, but execution skew increases and slave processes become idle
Solution Approach 1:
The system dynamically adjusts work granule distribution based on runtime pruning results. The query coordinator monitors which slave processes have completed their assigned granules and dynamically assigns additional granules to maintain balanced utilization, transforming the static initial distribution into a dynamic adaptive process that responds to actual execution progress
Solution Approach 2:
The system implements a feedback mechanism where the query coordinator receives completion status information from slave processes and uses this feedback to make informed decisions about dynamic work granule assignment. This closed-loop control ensures that workload distribution adjustments are based on actual execution state rather than static predictions
2Loss of time
If bloom partition pruning prunes a large portion of the workload, then processing time is reduced for remaining granules, but available slave processes are left idle with no work
Solution Approach 1:
The system performs preliminary assignment of additional work granules to slave processes that have completed their initial assignments, before those processes become completely idle. The query coordinator proactively pushes out new granules based on completion notifications, ensuring continuous work availability and preventing idle time
Solution Approach 2:
The system recovers unused computational resources by reassigning work granules to idle slave processes. When pruning eliminates certain work granules, the system recovers the computational capacity of processes that would otherwise be idle by assigning them new granules from the remaining workload, effectively recovering potential productivity loss
3Speed
If initial work granules are assigned before runtime pruning, then query execution can begin immediately, but workload distribution becomes unbalanced after pruning
Solution Approach 1:
The system transitions from a static initial workload distribution to a dynamic adaptive distribution model. The query coordinator continuously monitors execution progress and adjusts granule assignments in real-time, allowing the system to maintain both quick start (through initial assignment) and stable balanced distribution (through dynamic adjustments)
Solution Approach 2:
The system implements periodic rebalancing of work granules based on execution progress. Rather than continuous adjustment, the query coordinator periodically evaluates completion status and redistributes granules at appropriate intervals, balancing the overhead of monitoring against the benefit of maintained workload stability
Data Source
AI summary
Techniques herein improve computational efficiency for parallel queries with run-time data pruning by using adaptive granule generation. In an embodiment, an execution plan is generated for a query to be executed by a plurality of slave processes, the execution plan comprising a plurality of plan operators. For a first plan operator of the plurality of plan operators, a first set of work granules is generated, and for a second plan operator of the plurality of plan operators, a second set of work granules is generated. A first subset of slave processes of the plurality of slave processes is assigned the first set of work granules. Based on the execution of the first set of work granules by the first subset of slave processes, a bloom filter is generated that specifies for which of said first set of work granules no output rows were generated. Based on the bloom filter, the second set of work granules is modified and the modified second set of work granules is assigned to a second subset of slave processes and executed.