Distributed Token Bucket Rate Limiting for Multi-Controller Storage QoS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Implementing and enforcing quality of service (QoS) or rate limiting parameters for I/O streams across multiple storage controllers is complicated and impractical due to the overhead associated with messaging, which negates the benefits of using multiple controllers.
Innovation Solution
A method is introduced to manage I/O operations by determining target I/O rates for each storage controller, calculating local token bucket rates, and implementing a token bucket algorithm to enforce QoS attributes across multiple storage controllers, ensuring that each controller maintains a token bucket and distributes tokens randomly among supported logical volumes to prevent a capture effect.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a central controller is used to coordinate rate limiting across multiple storage controllers, then QoS enforcement becomes possible, but the messaging overhead negates the benefits of using multiple controllers
Solution Approach 1:
The patent divides the rate limiting function into independent segments at each storage controller. Each controller maintains its own token bucket and enforces rate limits locally without requiring centralized coordination. This segmentation eliminates the messaging overhead while preserving QoS enforcement capability across the distributed storage system.
Solution Approach 2:
Each storage controller autonomously manages its own rate limiting through local token bucket algorithms. The controllers self-regulate their I/O processing rates without external intervention or communication, thereby eliminating the need for complex inter-controller messaging while maintaining overall system QoS requirements.
2Device complexity
If token bucket rate limiting is applied at each storage controller independently, then messaging overhead is eliminated, but coordinating I/O rates across controllers becomes complicated
Solution Approach 1:
The patent transforms the rate limiting parameters into local token bucket rates that are derived from the overall I/O rate target. By changing the parameter representation from centralized I/O rates to distributed token bucket rates, each controller can independently regulate its I/O flow while collectively achieving the desired overall rate limit without complex coordination mechanisms.
3Reliability
If I/O rate limiting is enforced across multiple storage controllers, then QoS attributes are maintained, but the system becomes impractical due to coordination complexity
Solution Approach 1:
The patent segments the rate limiting enforcement into independent local token buckets at each storage controller. This segmentation allows QoS attributes to be enforced across the distributed system without requiring complex inter-controller coordination, thereby making the system practical to implement while maintaining reliability.
Solution Approach 2:
Each storage controller autonomously enforces QoS attributes through its own token bucket mechanism. This self-service approach eliminates the need for complex centralized coordination infrastructure, making the system easier to implement and maintain while ensuring QoS compliance across all controllers.
Data Source
AI summary
An I/O management method includes determining local target I/O rates for a plurality of storage controllers across which an I/O stream is distributed. The local target I/O rates may reflect a target latency corresponding to a QoS attribute associated with the I/O stream. A cumulative target I/O rate may be determined based on the local target I/O rates. A token bucket rate may be calculated for each applicable storage controller based on the local target I/O rates. The storage controllers may implement token bucket algorithms in accordance with the token bucket rates to enforce the QoS for the I/O stream. The local target I/O rates may be determined periodically based on measured data indicating actual latency and traffic. A queue depth is determined based on the measured values of latency and traffic. The queue depth is then used with the desired value of latency to obtain the target I/O rates.