Remote Shuffle Storage with Dynamic RSS Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In MapReduce systems, the load balancing between remote shuffle service (RSS) nodes is poor, leading to inefficient utilization of storage performance and potential overload of individual nodes.
Innovation Solution
A dynamic load balancing strategy is implemented by extending RSS nodes based on load pressure, allowing for real-time updates of load balancing strategies during the application's execution, and preconfiguring strategies for different stages to optimize data distribution across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a preset load balancing strategy is used for storing shuffle data in RSS nodes, then the application can be executed in the MapReduce system, but the load balancing between different RSS nodes is poor and storage performance cannot be fully utilized
Solution Approach 1:
The patent implements dynamic load balancing by allowing RSS nodes to be extended during application execution. The load balancing strategy is updated in real-time based on current node load conditions, transforming the static preset strategy into a dynamic adaptive system that responds to changing workload conditions.
Solution Approach 2:
The patent changes the parameter of load balancing strategy from fixed to variable. By introducing the concept of extending RSS nodes and updating strategies during execution, the system adapts parameters (node selection, data distribution) based on actual load conditions, improving both reliability and productivity.
2Device complexity
If shuffle data is stored according to a same load balancing strategy throughout the entire running process, then the implementation is simple, but a single RSS node may experience excessively high load pressure
Solution Approach 1:
The patent applies preliminary action by preconfiguring multiple load balancing strategies corresponding to different stages of the application. This allows the system to prepare appropriate strategies in advance for different execution phases, enabling smooth transitions without complex real-time decision-making during runtime.
Solution Approach 2:
The patent segments the application running process into multiple stages, each with its own load balancing strategy. This segmentation allows different strategies to be applied to different phases of data processing, preventing any single node from being overloaded throughout the entire process while maintaining manageable complexity.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
Embodiments of this application disclose a shuffle data storage method and apparatus, and a storage medium, and belong to the field of storage technologies. The method includes: An execution unit stores first shuffle data into at least one remote shuffle service RSS node according to a first load balancing strategy; the execution unit obtains an extended RSS node of a first RSS node; and the execution unit updates the first load balancing strategy to obtain a second load balancing strategy, to store second shuffle data according to the second load balancing strategy. Therefore, in a running process of a target application, the execution unit may obtain an extended RSS node for an RSS node, to update a load balancing strategy, so as to avoid storing shuffle data according to a same load balancing strategy in the entire running process of the target application. This can avoid excessively high load pressure of a single RSS node.