Runtime Data Skew Handling via Node Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big Data frameworks face significant execution time delays and potential failure due to data skew, which existing solutions cannot effectively address at runtime, especially in large-scale distributed computing environments with varying data distributions.
Innovation Solution
A system comprising a cluster of computing nodes with a processor that splits input data into partitions, detects data skew, and assigns skewed partitions to high-performance nodes with faster storage, while assigning non-skewed partitions to regular nodes for parallel processing, utilizing NVMe, Optane SSD, or persistent memory for enhanced performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data partitions are assigned to regular nodes with standard storage, then system cost is reduced, but execution time increases due to data skew
Solution Approach 1:
The patent applies local quality by differentiating node configurations based on their assigned data partitions. Second nodes are equipped with high-speed storage specifically for handling skewed partitions, while first nodes use standard storage for normal partitions. This localized optimization resolves the contradiction by improving execution time for critical skewed data without requiring all nodes to have complex high-performance configurations.
Solution Approach 2:
The patent segments the computing cluster into two types of nodes (first nodes with standard storage and second nodes with high-speed storage) and segments data partitions into skewed and non-skewed categories. This segmentation allows the system to match appropriate node types to appropriate data types, resolving the contradiction between execution time and device complexity by applying high-performance resources only where necessary.
2Productivity
If high-speed storage is deployed on all nodes, then processing speed increases, but system cost increases significantly
Solution Approach 1:
The patent implements partial action by deploying high-speed storage on only a subset of nodes (second nodes) rather than all nodes. The system dynamically identifies skewed partitions and assigns them to nodes with high-speed storage, achieving improved processing speed for critical data while using high-speed storage resources partially and efficiently, thus resolving the contradiction between productivity and resource quantity.
Solution Approach 2:
The patent changes the parameter of storage speed selectively based on data characteristics. Instead of uniformly upgrading all nodes, the system identifies partitions with data skew and assigns them to nodes with high-speed storage parameters, while leaving other nodes with standard storage. This parameter change approach resolves the contradiction by achieving high processing speed where needed without proportionally increasing overall resource consumption.
3Reliability
If data skew is not addressed, then system simplicity is maintained, but application reliability deteriorates
Solution Approach 1:
The patent applies preliminary action by detecting data skew in partitions before assigning them to nodes. The system analyzes data distribution characteristics and identifies skewed partitions in advance, then assigns them to appropriate second nodes with high-speed storage. This preliminary detection and classification mechanism ensures application reliability by preventing execution failures before they occur, while managing complexity through automated detection algorithms rather than complex manual data distribution management.
Data Source
AI summary
A system for handling data skew includes a cluster of computing nodes and a processor. The cluster includes one or more first nodes, each of which has a storage, and one or more second nodes, each of which has a storage. The storage of the respective second nodes has a higher access speed than the storage of the respective first nodes. The processor is configured to split input data into partitions of the input data, to detect if any of the partitions has data skew, and to assign ones of the partitions, which are detected as having no data skew, to the first nodes, and ones of the partitions, which are detected as having the data skew, to the second nodes, for parallel processing.