Big Data Reduce Processing Load Balancing via Key-Value Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big-data processing methods lead to load imbalance among reduce processing modules due to the uneven distribution of key-value pairs, resulting in inefficient processing.
Innovation Solution
A method and apparatus that assign key-value pairs to reduce processing modules based on the total value of each key-value pair set and the load status of each module, using modulo operations to group pairs with the same remainder and distribute them accordingly, ensuring more balanced loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If all key-value pairs are assigned to reduce processing modules according to keys using traditional modulo operation, then the distribution is simple to implement, but load imbalance occurs among reduce processing modules
Solution Approach 1:
The patent segments the key-value pairs into multiple groups based on remainder categories (first remainder group, second remainder group, etc.) and assigns different groups to different reduce processing modules. This segmentation ensures that modules with fewer tasks receive more key-value pairs, balancing the load while maintaining implementation simplicity through category-based distribution.
2Productivity
If key-value pairs are evenly distributed among reduce processing modules, then load balance is improved, but the complexity of the assignment mechanism increases
Solution Approach 1:
The patent changes the assignment parameter from simple modulo-based key hashing to a two-stage process: first grouping by remainder categories, then distributing groups to modules based on current load status. This parameter change achieves load balance while controlling complexity through the use of remainder categories as intermediate grouping units.
Solution Approach 2:
The patent implements a feedback mechanism where the system continuously monitors the number of tasks in each reduce processing module and uses this information to dynamically adjust the assignment of key-value pair groups. Modules with fewer tasks receive priority assignment, creating a self-balancing system that adapts to changing load conditions.
Data Source
AI summary
A big-data processing method is provided. The method includes obtaining multiple key-value pair sets, where each of the key-value pair sets includes at least one key-value pair, and remainders of modulo operations on all key-value pairs in each key-value pair set are the same, calculating a sum of all values included in each key-value pair set to obtain a total value of the values included in each key-value pair set, obtaining a load status of each reduce processor in a reduce processor set, assigning a corresponding key-value pair set to each reduce processor according to the total value of the values included in each key-value pair set and the load status of each reduce processor, and processing the assigned key-value pair set using each reduce processor.