Storage Node IO Aggregation Under Variable Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems using erasure codes (EC) algorithms face performance issues with small random read/write operations, particularly when using fixed aggregation policies, leading to memory overflow and reduced overall system performance due to varying bandwidth demands and node faults.
Innovation Solution
Adaptive adjustment of aggregation policies based on memory performance parameters, including bandwidth, fault information, and data access characteristics to aggregate small IO requests into larger requests, optimizing storage node operations and reducing memory overflow.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a fixed aggregation policy is used to aggregate small IO requests, then the system can maintain simple operation, but the memory may be quickly full under varying bandwidth demands and fault conditions, reducing system performance
Solution Approach 1:
The patent implements dynamic adjustment of aggregation policy parameters based on real-time memory performance parameters. The aggregation waiting time, aggregation threshold, and aggregation stripe size are adjusted dynamically according to memory bandwidth, memory capacity, and fault conditions, transforming the static fixed policy into a dynamic adaptive policy that responds to changing system conditions
Solution Approach 2:
The system continuously monitors memory performance parameters including bandwidth, capacity, and fault status, and uses this feedback to adjust aggregation policy parameters. The management node collects performance data from storage nodes and dynamically modifies aggregation parameters based on the feedback loop, enabling the system to adapt to varying load conditions and fault scenarios
2Productivity
If the aggregation waiting time is increased to aggregate more small IO requests, then the data processing efficiency is improved, but the memory may overflow under high input bandwidth conditions
Solution Approach 1:
The patent changes the aggregation waiting time parameter dynamically based on memory bandwidth conditions. When memory bandwidth is high and input requests are intensive, the aggregation waiting time is reduced to prevent memory overflow. When bandwidth is lower and request intensity is moderate, the waiting time is increased to improve data processing efficiency by aggregating more requests
3Quantity of substance
If a larger EC ratio is used to improve disk rate, then the available capacity per unit storage space increases, but the performance of small random read/write operations deteriorates due to increased splitting into small operations
Solution Approach 1:
The patent applies preliminary aggregation of small IO requests into larger requests before they are processed by the EC algorithm. By pre-aggregating requests at the storage node memory, the system reduces the number of small random operations that reach the EC processing stage, thereby mitigating the performance deterioration caused by high EC ratios while maintaining the capacity benefits
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
This application provides a data processing method, including: obtaining a parameter related to performance of a memory of a storage node, adjusting an aggregation policy based on the parameter, aggregating a first input output IO request in the memory into a second IO request based on an adjusted aggregation policy, and performing persistent storage on data corresponding to the second IO request. On one hand, the storage node aggregates small IO requests into a large IO request, so as to avoid impact of the small IO requests on performance of an entire distributed storage system. On the other hand, the aggregation policy is adaptively adjusted, it can be avoided that the memory is quickly full due to aggregation performed based on an aggregation policy with a fixed parameter, thereby affecting overall performance of a distributed storage system.