Cluster Storage Load Balancing via Proximity Data Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale cloud storage systems face load imbalance issues due to diverse I/O workloads, leading to hot spots and high delays, with existing solutions either ineffective at large scales or costly in terms of reconfiguration.
Innovation Solution
A method and system for load balancing in cluster storage systems that identifies a source node with excessive load, selects data objects to move based on load values, and migrates them to a target node within a predetermined proximity, ensuring the source node's load remains below a threshold, thereby alleviating hot spots and reducing reconfiguration costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is shuffled across the cluster to balance load, then load balance is improved, but bandwidth cost increases
Solution Approach 1:
The patent applies local quality by prioritizing data migration to nearby nodes within the same rack or data center before considering distant nodes. This hierarchical approach migrates data locally first (same rack), then to the same data center, and finally across data centers only when necessary, thereby improving load balance while minimizing bandwidth consumption by preferring local transfers over long-distance transfers.
2Loss of energy
If offline optimization solvers are used to minimize reconfiguration costs, then reconfiguration cost is reduced, but computational expense increases
Solution Approach 1:
The patent segments the data migration problem into two distinct phases: an offline planning phase that uses optimization solvers to determine the optimal migration plan, and an online execution phase that simply follows the pre-computed plan. This segmentation allows the computationally expensive optimization to be performed once offline, while online operations remain lightweight, thus reducing overall reconfiguration costs without excessive computational burden during runtime.
Solution Approach 2:
The patent applies preliminary action by performing the computationally expensive optimization computation offline before actual data migration occurs. The offline solver pre-determines the optimal migration plan, which is then executed during online operations without requiring real-time computation. This approach minimizes online computational expense while still achieving optimal reconfiguration cost reduction.
3Ease of operation
If data objects are migrated to achieve load balance, then hot spots are reduced, but reconfiguration expense increases
Solution Approach 1:
The patent applies partial action by migrating only the necessary portion of data objects required to eliminate hot spots, rather than performing complete cluster-wide rebalancing. The system identifies specific overloaded nodes and migrates just enough data to bring their load below thresholds, avoiding unnecessary migration of already-balanced data. This selective approach reduces reconfiguration expenses while effectively reducing hot spots.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and systems for load balancing in a cluster storage system are disclosed herein. The method includes identifying a source node within the cluster storage system from which to move a number of data objects, wherein the source node includes a node with a total load exceeding a threshold value. The method also includes selecting the data objects to move from the source node, wherein the data objects are chosen such that the total load of the source node no longer exceeds the threshold value. The method further includes determining a target node within the cluster storage system based on a proximity to the source node and the total load of the target node and moving the data objects from the source node to the target node.