Multi-Layer Load Balancing in Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cluster-wide load balancing across storage arrays is challenging due to asymmetry in application affinity and configuration, leading to underperformance when only local node-level load balancing is performed, as the storage array does not serve as an application host.
Innovation Solution
A computer-implemented method for load balancing that discovers multiple hosts in a cluster, collects global I/O load characteristics, and selects a host based on current load conditions to redirect I/Os, effectively balancing loads across multiple layers including multi-pathing and volume management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If local node-level load balancing is performed, then device complexity is reduced, but storage system performance deteriorates due to inability to balance loads cluster-wide
Solution Approach 1:
The load balancing functionality is segmented into multiple layers: local node-level multi-pathing layer and cluster-wide volume management layer. Each layer handles load balancing independently at its respective scope, allowing local simplicity while achieving global optimization through the volume manager's cluster-wide perspective
Solution Approach 2:
The volume manager acts as an intermediary between the local multi-pathing layer and the storage array. It collects global I/O load characteristics from multiple hosts and makes intelligent I/O routing decisions, mediating between local load balancing limitations and cluster-wide performance optimization needs
2Productivity
If cluster-wide load balancing is performed at the storage array, then storage system performance is improved, but device complexity increases and configuration asymmetry problems arise
Solution Approach 1:
The cluster-wide load balancing functionality is extracted from the storage array and placed in the host's volume manager. This removes the complexity of maintaining global load balance state from the storage array, while still achieving cluster-wide load balancing through host-based I/O redirection
Solution Approach 2:
Instead of having the storage array perform cluster-wide load balancing, the invention inverts the approach by having hosts perform load balancing decisions based on global characteristics collected by the volume manager. The storage array simply receives I/Os from the selected host without needing to understand cluster-wide load states
3Ease of operation
If only local node-level load balancing is performed, then configuration asymmetry is avoided, but productivity decreases due to underperforming cluster utilization
Solution Approach 1:
The invention adds a new dimension to load balancing by operating at the volume management layer rather than just the multi-pathing layer. This higher-level dimension enables cluster-wide visibility and I/O routing decisions while maintaining the simplicity of local configuration through the volume manager's centralized control
Data Source
AI summary
A method and system for load balancing. The method includes discovering each of a plurality of hosts in a cluster, wherein the plurality of hosts is configured for accessing a LUN of a storage system through a storage network fabric. Global input/output (I/O) load characteristics are collected for each of the plurality of hosts at the device and/or volume level. A selected host is determined for processing an I/O originating at the local host, wherein the host is selected based on a current set of the global I/O load characteristics.