Storage Cluster Load Balancing via Dynamic IOPS Regulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face issues of hot spots where some drives are over-utilized while others are under-utilized, leading to inconsistent client performance due to uneven data distribution, which existing quality of service prioritization methods fail to fully address.
Innovation Solution
A performance management system that determines a target performance value based on previous client performance and cluster health, regulating access to ensure even load distribution across all drives by adjusting the number of IOPS or other performance metrics, thereby optimizing data access and preventing hot spots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored on a small percentage of drives in a storage cluster, then storage capacity is utilized, but hot spots occur where portions of the cluster are over-utilized while other portions are under-utilized
Solution Approach 1:
The patent segments the storage cluster into multiple portions and distributes client data across these segments. By stripe data across multiple drives and monitor performance metrics for each drive, the system divides the storage resource into manageable units that can be independently monitored and balanced, preventing hot spots from forming on any single drive or portion of the cluster.
Solution Approach 2:
The patent applies local quality by monitoring and regulating performance metrics for each individual drive or storage portion separately. The system tracks read/write IOPS, latency, and throughput for each drive independently, then applies performance regulation specifically to drives that are over-utilized, allowing each portion of the cluster to have optimized characteristics based on its actual performance needs.
2Ease of operation
If quality of service based on client prioritization is implemented, then client experience is slightly improved, but specific consistent performance levels are not guaranteed
Solution Approach 1:
The patent implements feedback by continuously monitoring performance metrics for each client and each drive, then using this information to dynamically regulate client performance. The system measures actual read/write IOPS, latency, and throughput, compares these against target values, and adjusts client access accordingly. This closed-loop feedback mechanism ensures that clients receive consistent performance levels regardless of cluster load conditions.
Solution Approach 2:
The patent applies dynamics by making performance regulation adaptive and changeable in real-time. Rather than static prioritization, the system dynamically adjusts client performance targets and regulation based on current cluster conditions, drive performance, and client needs. This allows the system to respond to changing conditions while maintaining guaranteed performance levels for each client.
Data Source
AI summary
In one embodiment, a method includes determining a previous client performance value in terms of a performance metric for a volume in a storage system. The previous client performance value is related to previous access for a client to the volume. Also, the storage system is storing data for a plurality of volumes where data for each of the plurality of volumes is striped substantially evenly across drives of the storage system. The method applies criteria to the previous performance value to determine a target performance value. Performance of the client with respect to access to the volume is regulated in terms of the performance metric based on the target performance value.