Server Mapping Plans for Data Skew in Parallel Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel execution techniques in database systems suffer from data skew issues during server mapping, leading to performance degradation due to uneven distribution of workloads among server groups, especially in cases where the number of partitions is smaller than the possible degree of parallelism, resulting in idle groups or overloaded servers.
Innovation Solution
Implement compile-time and runtime solutions to detect skew by adjusting server mapping, using dynamic sampling queries and partition mapping vectors to redistribute workload evenly among server processes, and switching to alternative distribution methods when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If server mapping is used to increase degree of parallelism, then parallelism is improved, but data skew causes uneven workload distribution leading to performance degradation
Solution Approach 1:
The patent implements dynamic server mapping where the mapping between partitions and server processes is not fixed but can be adjusted at runtime. The system monitors partition sizes and dynamically remaps partitions to server processes to balance workload distribution, resolving the contradiction between maintaining high parallelism and ensuring uniform workload distribution.
Solution Approach 2:
The system changes the mapping parameters dynamically based on detected partition skew. When skew is detected, the system modifies the server mapping parameters to redistribute partitions more evenly across server processes, thereby maintaining both high parallelism and workload uniformity.
2Device complexity
If static server mapping is used, then implementation is simple, but performance degrades when partition sizes vary
Solution Approach 1:
The patent transitions from static to dynamic server mapping, allowing the system to adapt to varying partition sizes at runtime. This dynamic approach maintains simplicity in implementation while significantly improving performance under skewed partition conditions.
Solution Approach 2:
The system incorporates feedback mechanisms where partition size information is collected and used to adjust server mapping. This feedback loop enables the system to maintain optimal performance by adapting to actual partition sizes without complicating the basic mapping implementation.
3Productivity
If partitions are sub-partitioned to increase parallelism, then degree of parallelism is improved, but overhead increases and customers may prefer hash-based operations
Solution Approach 1:
Instead of requiring customers to manually sub-partition data, the system dynamically adjusts server mapping based on existing partition sizes. This eliminates the need for additional sub-partitioning operations while achieving high parallelism, thereby reducing overhead and avoiding the complexity of hash-based redistribution.
Data Source
AI summary
Skew handling techniques are provided in parallel execution for even load balancing and scaling. In a compile-time solution, a dynamic sampling query is issued to detect partition skew. The compile-time solution determines the number of skewed partitions and uses a hybrid distribution scheme where skewed partitions use a random distribution and non-skewed partitions use the original server mapping. In a runtime solution, producer server processes create partition mapping vectors that contain partition mapping information. Each producer server process sends its partition mapping vector to the query coordinator (QC). The QC receives the partition mapping vectors from the producer server processes, merges the vectors, and determines a skew result based on the merged mapping vectors and sends the skew result to the producer server processes. The producer server process can alter distribution of skewed partitions based on the skew result.