Data-Aware Parallel Rollup Execution for Database Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel execution schemes for rollup operations in relational database management systems face scalability issues due to low cardinality of non-rollup columns and data explosion caused by sparse data distribution, leading to inefficient performance and resource overload.
Innovation Solution
A data-aware parallel execution scheme dynamically selects a data distribution key from candidate keys based on monitoring distinct values and skewedness, allowing for adaptive compression and distribution of data between rollup distributors and collectors, optimizing data distribution and computation across multiple processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If parallel execution schemes are used for rollup operations, then processing speed is improved, but scalability deteriorates due to low cardinality of non-rollup columns
Solution Approach 1:
The patent implements dynamic parallel execution by monitoring the cardinality of non-rollup columns at runtime and adjusting the number of parallel processes accordingly. The system starts with an initial number of parallel processes and dynamically increases or decreases the count based on the actual data characteristics, allowing the execution scheme to adapt to different data scenarios and maintain both speed and scalability.
2Productivity
If data is distributed across multiple processes, then processing efficiency is improved, but data explosion occurs due to sparse data distribution
Solution Approach 1:
The patent applies local quality by distributing data to parallel processes based on the actual data distribution characteristics. Instead of uniform distribution, the system identifies regions with sufficient data density and directs processing to those areas, while avoiding distribution to regions with sparse data. This localized approach ensures processing efficiency is improved without causing data explosion in underutilized processes.
3Ease of manufacture
If fixed number of parallel processes is used, then implementation simplicity is maintained, but performance optimization is limited due to unknown data characteristics
Solution Approach 1:
The patent implements a feedback mechanism where the system monitors data characteristics such as the cardinality of non-rollup columns during execution. Based on this feedback, the system dynamically adjusts the number of parallel processes to optimize performance. The feedback loop continues throughout execution, allowing the system to adapt to actual data distribution patterns and achieve optimal performance without complex pre-analysis.
Data Source
AI summary
According to one aspect of the invention, for a database statement that specifies rollup operations, a data distribution key may be selected among a plurality of candidate keys. Numbers of distinct values of the candidate keys may be monitored with respect to a particular set of rows. Hash values may also be generated by column values in the candidate keys. The data distribution key may be determined based on results of monitoring the numbers of distinct values of the candidate keys as well as the frequencies of hash values computed based on column values of the candidate keys. Rollup operations may be shared between different stages of parallel executing processes and data may be distributed between the different stages of parallel executing processes based on the selected data distribution key.