Dynamic Partition Reassignment for Big Data Skew
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Big data analytics on Hadoop Distributed File System (DFS) often experiences performance degradation due to data and processing skews, where worker tasks execute at varying times due to uneven data distribution and heterogeneous computing resources, leading to resource underutilization and delayed data synchronization.
Innovation Solution
A method and system for workload balancing that uses cost-based static partitioning to assign data partitions to reducers, with a dynamic partitioning approach that allows for reassignment of partitions to balance the load, ensuring all reducers process data uniformly, and maintains an unassigned pool for dynamic allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If worker tasks are executed in distributed fashion on cluster nodes, then data processing capability is improved, but execution time varies significantly due to data skew and processing skew
Solution Approach 1:
The patent implements dynamic partition assignment where the system transitions from static partition distribution to dynamic adjustment based on real-time task execution status. The framework monitors task progress and reassigns partitions from completed tasks to pending tasks, enabling the system to adapt to actual processing speeds and reduce idle waiting time across cluster nodes.
Solution Approach 2:
The patent employs feedback mechanisms by continuously monitoring task execution progress and using this information to make informed partition reassignment decisions. The system tracks which partitions have been processed and which tasks are still pending, then uses this feedback to dynamically balance the workload, ensuring that faster-completing tasks can utilize additional partitions to maintain high utilization.
2Quantity of substance
If more worker tasks are created to process large data volumes, then processing thoroughness is improved, but resource utilization decreases due to skew
Solution Approach 1:
The system dynamically adjusts the distribution of partitions to tasks based on real-time execution status. Instead of statically assigning partitions, the framework allows partitions to be reassigned from completed tasks to pending tasks, ensuring that computing resources are continuously utilized and no node remains idle while others are overloaded.
Solution Approach 2:
The patent changes the parameter of partition assignment from fixed to variable. The system monitors execution progress and modifies partition assignments dynamically, allowing the same set of partitions to be redistributed multiple times during job execution based on actual processing speeds and task completion status.
3Device complexity
If static partition assignment is used, then system simplicity is maintained, but performance degrades due to inability to handle skew
Solution Approach 1:
The patent introduces dynamic partition assignment that builds upon the existing static assignment framework. The system maintains the simplicity of initial static partition distribution but adds a dynamic layer that monitors task progress and performs reassignments when beneficial, thus preserving system simplicity while significantly improving performance through adaptive workload balancing.
Solution Approach 2:
The system performs preliminary static partition assignment to establish initial task workloads, then adds dynamic reassignment capability that activates when performance degradation is detected. This approach maintains the benefits of simple initial setup while incorporating performance optimization when needed.
Data Source
AI summary
Data partitions are assigned to reducer tasks using a cost-based and workload balancing approach. At least one of the initial data partitions remains unassigned in an unassigned partitions pool. Each reducer while working on its assigned partitions makes dynamic run-time decisions as to whether to: reassign a partition to another reducer, accept a partition from another reducer, select a partition from the unassigned partitions pool, and/or reassign a partition back to the unassigned partitions pool.