MapReduce Data Reallocation via Helper Reduce Processes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
MapReduce processes often suffer from load imbalances among reduce processes due to uneven distribution of intermediate data, leading to delayed completion times as heavily loaded processes take longer to finish than lightly loaded ones, making it difficult for users to manage and balance loads across multiple computing platforms.
Innovation Solution
Intermediate data is reallocated based on load factors associated with reduce processes, using helper reduce processes to distribute and process data more evenly, thereby reducing completion times and balancing loads across multiple reduce processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If intermediate data is distributed among reduce processes using traditional MapReduce allocation, then the system can process data in parallel, but load imbalances occur causing delayed completion times
Solution Approach 1:
The patent implements dynamic load balancing by allowing reduce processes to request additional intermediate data allocations based on their current load status. Helper reduce processes are dynamically created or activated to receive reallocated data from overloaded reduce processes, enabling the system to adapt to changing load conditions during execution rather than using static initial allocation
Solution Approach 2:
The patent introduces intermediary mechanisms including load factor calculations that mediate between data distribution and processing capacity. The system uses intermediary helper reduce processes that act as buffers to receive reallocated data from overloaded processes and complete processing, thereby mediating the load imbalance problem
2Productivity
If more reduce processes are used to increase processing capacity, then productivity improves, but load management complexity increases making it difficult for users to balance loads
Solution Approach 1:
The patent enables reduce processes to self-manage their load by automatically calculating load factors and requesting reallocations when overloaded. The system provides self-service load balancing where processes monitor their own status and initiate reallocation requests without requiring complex external management or user intervention
Solution Approach 2:
The patent changes the parameter of data allocation from fixed to variable based on load factor calculations. The system dynamically adjusts the amount of intermediate data allocated to each reduce process based on real-time load conditions, using parameter changes in allocation ratios and process activation states to maintain balance
Data Source
AI summary
The subject matter disclosed herein relates to a system and/or method for allocating data among reduce processes. In a particular implementation, a map process may be executed to provide intermediate data associating key/value pairs with input data. Intermediate data may be allocated among a plurality of reduce processes. At least a portion of intermediate data initially allocated to one or more of said reduce processes may be re-allocated based, at least in part, on a load factor associated with one or more reduce processes.