Database Skew Reduction via Unique Distribution Attributes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In file systems with multiple processing modules, uneven distribution of rows in large relations during join or aggregation operations leads to parallel inefficiency and longer processing times due to skew, where some modules have significantly more rows to process than others.
Innovation Solution
A method is implemented to reduce skew by setting a distribution attribute in each relation entry to a unique value and redistributing the entries among processing modules based on this attribute, and creating multiple copies of the second relation to ensure balanced processing across modules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If rows are distributed based on hash redistribution or local storage, then processing modules can operate in parallel, but skew in row distribution causes imbalance and reduces parallel efficiency
Solution Approach 1:
The patent applies preliminary action by pre-distributing data rows to processing modules in a balanced manner before the join or aggregation operation begins. A distribution column is added and populated with values that ensure even row distribution across modules, preventing skew before it occurs and enabling all modules to work in parallel at full efficiency throughout the operation.
Solution Approach 2:
The patent changes the distribution parameter by introducing a distribution column with specifically chosen values that transform the original skewed distribution into a balanced one. This parameter change ensures that each processing module receives a comparable number of rows, allowing parallel operations to proceed without idle modules waiting for others to complete.
2Productivity
If a distribution column is added to reduce skew, then row distribution becomes more balanced, but data structure complexity increases
Solution Approach 1:
The patent uses copying by creating a distribution column that contains reference values from an existing column in the data. Rather than storing complex distribution logic, the system copies relevant identifier values into the distribution column, providing a simple mechanism for balanced distribution that maintains data structure simplicity while achieving the desired load balancing effect.
Data Source
AI summary
A database operation is performed in a file system residing on a plurality of processing modules. The file system includes a first relation having a plurality of first-relation entries. Each of the plurality of first-relation entries has a first-relation attribute that is of interest in the database operation. A value of a distribution attribute in each of the first-relation entries is set to a unique value selected from among a domain of unique values. The first-relation entries of the first relation are redistributed among the plurality of processing modules based on the first-relation attribute and the distribution attribute. The computational operation is performed to produce a result.