Dynamic Flow Control for Storage Clusters with Latency Guarantees
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in managing input/output (I/O) requests efficiently, leading to potential overloading and increased latency due to varying patterns of I/O operations, which can result in cluster instability and degraded performance.
Innovation Solution
Implementing dynamic flow control through a dynamic bandwidth limiter that monitors and adjusts the choker threshold based on average and peak latency, queuing excessive I/O requests to prevent overload and maintain latency within configurable thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the storage system processes all I/O requests without flow control, then data throughput is maximized, but cluster overload occurs and latency increases
Solution Approach 1:
The choker threshold is made dynamic rather than static. The system continuously monitors I/O latency and adjusts the choker threshold accordingly - increasing it when latency is low to allow more I/O requests through, and decreasing it when latency is high to prevent overload. This dynamic adjustment resolves the contradiction by adapting the flow control parameter to current system conditions, maintaining both high throughput and cluster stability.
Solution Approach 2:
The system implements a feedback mechanism where I/O latency is continuously monitored and used to adjust the choker threshold. The monitoring component tracks latency metrics, and this information feeds back to the flow control mechanism which adjusts the threshold to prevent cluster overload while maximizing throughput. This closed-loop control resolves the contradiction by using real-time system state information to balance throughput and stability.
2Reliability
If a fixed choker threshold is used for flow control, then cluster overload is prevented, but data throughput is reduced due to unnecessary queuing
Solution Approach 1:
The choker threshold transitions from a fixed value to a dynamic parameter that adjusts based on monitored I/O latency. When latency is low, the threshold increases to allow more I/O requests through, improving throughput. When latency is high, the threshold decreases to prevent overload, maintaining stability. This dynamic behavior resolves the contradiction by making the threshold adaptive rather than static.
Solution Approach 2:
The system changes the parameter (choker threshold) based on monitored conditions. Instead of using a constant threshold value, the system adjusts the threshold parameter dynamically according to I/O latency measurements. This parameter change allows the system to optimize both throughput and stability by adapting the flow control parameter to current workload conditions.
3Reliability
If I/O requests are queued frequently to maintain latency thresholds, then latency is controlled, but processing time increases
Solution Approach 1:
The choker threshold parameter is adjusted dynamically based on monitored I/O latency. When latency is within acceptable ranges, the threshold is increased to reduce queuing and minimize processing time. When latency approaches unacceptable levels, the threshold is decreased to control latency by queuing excess requests. This parameter adjustment resolves the contradiction by optimizing the balance between latency control and processing time based on real-time conditions.
Solution Approach 2:
The flow control mechanism dynamically adjusts the choker threshold based on current system state rather than using a static value. This dynamic adjustment reduces unnecessary queuing when the system can handle the load, minimizing processing time, while still providing latency guarantees when needed by increasing queuing under heavy load conditions.
Data Source
AI summary
Described embodiments may provide methods and systems for receiving one or more input/output (I/O) requests by a storage system having at least one storage cluster. The storage system performs each I/O request with dynamic flow control by determining a latency associated with the one or more received I/O requests during at least one monitoring interval and tracking I/O requests to the storage cluster. If a received I/O request exceeds a choker threshold value of the storage cluster, the I/O request is queued. Otherwise, the received I/O request is performed for the storage cluster.