Dynamic IO Shaping for Distributed Storage Node Failures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance degradation due to failure events, leading to IO processing delays and timeouts in other storage nodes, as conventional approaches fail to dynamically adjust to failures, impacting high availability and overall system performance.
Innovation Solution
Implementing a dynamic IO shaping mechanism that switches from an average latency mode to an individual latency mode in response to detected failure events, reducing the number of concurrent IO operations by generating and sending backpressure hints across storage nodes, thereby enhancing responsiveness and stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional IO shaping mechanisms are used that rely on average latency over multiple IO operations, then normal IO processing can be maintained under stable conditions, but the system experiences performance degradation and timeouts when failure events occur
Solution Approach 1:
The IO shaping mechanism dynamically switches between two operational modes: a first mode using average latency over multiple IO operations under normal conditions, and a second mode using latencies of individual IO operations when failure events are detected. This dynamic adaptation allows the system to maintain stability during failures while preventing performance degradation
Solution Approach 2:
The system continuously monitors IO operation latencies and uses this feedback to detect failure events. When individual IO latencies exceed thresholds, the system triggers a mode switch in the IO shaping mechanism, adjusting the number of concurrent IO operations to prevent system overload and maintain reliability
2Productivity
If the number of concurrent IO operations is increased to improve throughput, then system productivity increases under normal conditions, but the system becomes vulnerable to overload and degradation during failure events
Solution Approach 1:
The IO shaping mechanism proactively reduces the number of concurrent IO operations when failure events are detected, before the system becomes overloaded. By anticipating potential system degradation and preemptively adjusting IO concurrency, the mechanism prevents timeouts and maintains stability during failures
Solution Approach 2:
The system changes key operational parameters of the IO shaping mechanism based on system state: switching from using average latency (first mode) to individual IO latencies (second mode) as the controlling parameter when failures occur. This parameter change enables the system to respond more aggressively to degradation while maintaining high throughput under normal conditions
Data Source
AI summary
At least one processing device is configured to detect a failure event impacting at least a first storage node of a plurality of storage nodes of a distributed storage system, and responsive to the detected failure event, to modify an input-output (IO) shaping mechanism in each of the storage nodes in order to at least temporarily reduce a total number of IO operations that are concurrently processed in the distributed storage system. For example, modifying an IO shaping mechanism in each of the storage nodes illustratively comprises transitioning the IO shaping mechanism in each of the storage nodes from a first operating mode to a second operating mode that is different than the first operating mode. The second operating mode of the IO shaping mechanism illustratively has a relatively faster responsiveness to changes in IO operation latency as compared to the first operating mode of the IO shaping mechanism.