Adaptive Window Function Evaluation in Parallel Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel execution schemes for evaluating reporting window functions in relational database management systems do not scale well, failing to fully leverage the system's parallel processing resources due to significant computation overheads and data traffic, especially when the number of distinct values for data distribution keys is low.
Innovation Solution
The proposed solution involves scalable computation algorithms that use an extended data-distribution key or computation pushdown strategy to distribute input rows among multiple processing entities, allowing for partial aggregations to be performed in parallel and then consolidated into full aggregations, while adaptively switching execution strategies based on runtime monitoring of distinct values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional parallel execution schemes are used for evaluating reporting window functions, then the system can process data in parallel, but it fails to scale well due to significant computation overheads and data traffic
Solution Approach 1:
The patent divides the evaluation of reporting window functions into multiple stages: a first stage that evaluates a first subset of reporting window functions, and a second stage that evaluates a second subset of reporting window functions. This segmentation allows the system to process different subsets of window functions in parallel without overwhelming computational overhead, as each stage can be independently optimized and executed by different processing entities.
Solution Approach 2:
The patent dynamically determines the evaluation stages based on the actual number of distinct values for partition-by keys of the reporting window functions. The system adapts its execution plan at runtime, switching between different evaluation strategies depending on the data characteristics. This dynamic approach allows the system to scale efficiently by adjusting the parallelization strategy based on actual workload conditions rather than using a fixed approach.
2Speed
If more parallel processing resources are utilized, then computation speed improves, but data traffic and computation overheads increase significantly
Solution Approach 1:
The patent merges the evaluation of multiple reporting window functions into a unified multi-stage process that shares common computations. By evaluating the first subset of reporting window functions in the first stage and the second subset in the second stage, the system avoids redundant computations that would occur if each window function were evaluated separately. This merging reduces data traffic and computation overheads while maintaining high parallel processing efficiency.
3Reliability
If the system evaluates all reporting window functions simultaneously, then complete results are obtained, but the computation overhead and complexity increase
Solution Approach 1:
The patent segments the complete set of reporting window functions into a first subset and a second subset, evaluating them in separate stages. This segmentation maintains computation completeness by ensuring all window functions are eventually evaluated, while reducing execution complexity by allowing each stage to be independently managed and optimized. The system can process the first subset through one set of operations and the second subset through another set of operations, simplifying the overall execution plan.
Data Source
AI summary
According to one aspect of the invention, for a database statement that specifies evaluating reporting window functions, a computation-pushdown execution strategy may be used for the database statement. The computation-pushdown execution plan includes producer operators and consolidation operators. Each producer operator computes a respective partial aggregation for each reporting window function based on a subset of rows, and broadcasts the respective partial aggregation. Each consolidation operator fully aggregates all partial aggregations broadcasted from the producer operators. Alternatively, an extended-data-distribution-key execution plan may be used. Each producer operator sends rows based on hash keys to sort operators for computing partial aggregations for at least one reporting window function based on a subset of rows. Each consolidation operator receives and fully aggregates all partial aggregations broadcasted from the sort operators.