Dynamic Shuffle Data Storage Across RSS Nodes Under Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In MapReduce systems, the existing load balancing strategies for storing shuffle data in remote shuffle service (RSS) nodes result in poor performance utilization and excessive load pressure on individual nodes, leading to bottlenecks and inefficient data distribution.
Innovation Solution
A dynamic load balancing strategy is implemented by updating the load balancing strategy based on extended RSS nodes during the execution of a target application, allowing for different strategies to be configured for various stages and adjusting in real-time to avoid overloading single nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a preset load balancing strategy is used to store shuffle data in RSS nodes, then the data storage process is simple and straightforward, but the load balancing performance is poor and storage performance cannot be fully utilized
Solution Approach 1:
The patent transforms the static preset load balancing strategy into a dynamic strategy that automatically adjusts during execution. The system monitors RSS node load in real-time and dynamically selects target RSS nodes for storing shuffle data, allowing the load balancing strategy to adapt to changing system conditions and fully utilize storage performance.
Solution Approach 2:
The patent implements a feedback mechanism where the system monitors the load status of RSS nodes and uses this information to adjust the load balancing strategy. The monitoring module provides feedback on node load, and the selection module uses this feedback to make informed decisions about where to store shuffle data, creating a closed-loop control system that optimizes storage performance.
2Device complexity
If a preset load balancing strategy is used throughout the entire execution process, then the strategy is simple to implement, but it causes excessive load pressure on single RSS nodes
Solution Approach 1:
The patent makes the load balancing strategy dynamic by allowing RSS node selections to change during execution. Instead of using a fixed preset strategy, the system dynamically adjusts target RSS nodes based on real-time load monitoring, preventing any single node from becoming overloaded while maintaining manageable complexity through automated decision-making.
Solution Approach 2:
The patent segments the data storage process into multiple stages with different load balancing strategies. The system divides the execution process and applies different RSS node selection criteria at different stages, which distributes load more evenly across nodes and prevents excessive pressure on any single node.
3Stability of the object's composition
If the same load balancing strategy is used for all map tasks, then the implementation is consistent and simple, but it creates bottlenecks in data distribution
Solution Approach 1:
The patent introduces dynamic adjustment of load balancing strategies based on execution stage and RSS node status. The system maintains consistency in its automated decision-making framework while adapting the specific strategy application to current conditions, eliminating data distribution bottlenecks through real-time optimization.
Solution Approach 2:
The patent changes key parameters of the load balancing strategy based on execution context. The system adjusts selection criteria and target node identification parameters dynamically during different stages of map task execution, optimizing data distribution efficiency without compromising the stability of the overall control framework.
Data Source
AI summary
A shuffle data storage method includes: an execution unit stores first shuffle data into at least one remote shuffle service RSS node according to a first load balancing strategy; the execution unit obtains an extended RSS node of a first RSS node; and the execution unit updates the first load balancing strategy to obtain a second load balancing strategy, to store second shuffle data according to the second load balancing strategy.