Distributed Storage Load Balancing via Virtual Node Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise storage systems face challenges with centralized models that make it difficult to scale capacity, performance, and cost independently, leading to imbalanced storage nodes and uneven workload distribution due to non-deterministic workloads, resulting in storage node overload and inefficient resource utilization.
Innovation Solution
A distributed storage system called StorFS performs automated load balancing by dynamically managing virtual nodes across physical nodes, using a Cluster Resource Manager to monitor resources and rebalance vNodes without disrupting user I/O operations, optimizing resource allocation and distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a centralized storage model is used, then storage controller functions are integrated into the same physical unit, but it becomes difficult to independently scale storage capacity, performance, and cost
Solution Approach 1:
The patent segments the storage system into distributed virtual nodes (vNodes) that can be independently managed and scaled across multiple physical nodes. This segmentation allows independent scaling of storage capacity and performance without requiring changes to the entire system architecture, directly resolving the contradiction between adaptability and device complexity.
2Quantity of substance
If storage nodes are distributed across multiple physical nodes, then storage capacity can be scaled, but load imbalance occurs where some nodes become full before others
Solution Approach 1:
The patent implements a load balancing mechanism that continuously monitors storage capacity and I/O workload across distributed nodes, detecting imbalances and automatically redistributing data and workloads. This feedback-driven approach maintains even utilization across all nodes despite varying storage capacities, resolving the contradiction between scaling quantity and operational ease.
Solution Approach 2:
The system dynamically adjusts data placement and workload distribution based on real-time node status. Virtual nodes can be migrated between physical nodes as conditions change, allowing the system to adapt to varying loads and maintain balance while scaling storage capacity, thus resolving the contradiction between quantity and ease of operation.
3Productivity
If load balancing is performed by moving virtual nodes between physical nodes, then storage utilization is optimized, but user I/O operations may be disrupted
Solution Approach 1:
The patent creates a copy of the virtual node on the destination physical node before migrating the original. This preliminary action ensures that data is replicated and available on both nodes during the transition, allowing I/O operations to continue uninterrupted while the virtual node is being moved, thus resolving the contradiction between productivity and reliability.
Solution Approach 2:
The patent introduces a copy mechanism as an intermediary during virtual node migration. The copy serves as a temporary mediator that maintains data availability and I/O continuity while the original virtual node is being relocated, enabling load balancing without disrupting user operations and resolving the contradiction between efficiency and continuity.
4Quantity of substance
If new storage appliances are added to datacenters, then storage capacity increases, but management overhead increases significantly
Solution Approach 1:
The patent creates a universal management layer that handles load balancing, data placement, and node migration across all physical nodes through a single distributed file system interface. This universal approach allows multiple storage appliances to be managed through unified software, reducing management overhead while increasing storage capacity and resolving the contradiction between quantity and device complexity.
Data Source
AI summary
A distributed storage system that performs automated load balancing is described. In an exemplary embodiment, a storage controller server determines if there is duplicative data in a distributed storage system. In this embodiment, the storage controller server detects a load balancing event in the distributed storage system, where the distributed storage system includes a plurality of virtual nodes distributed across a plurality of physical nodes. In response to detecting the load balancing event, the storage controller server determines that a current virtual node is to move from a source physical node to a destination physical node. In addition, the current virtual node is one of the plurality of virtual nodes and the source and destination physical nodes are in the plurality of physical nodes. The storage controller server further moves the current virtual node from the source physical node to the destination physical node.