Adaptive IO Aggregation for Memory-Aware Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face performance issues due to the use of fixed aggregation policies for IO requests, which can lead to memory overflow and reduced system performance when input/output bandwidth imbalances or node faults occur, especially with high EC ratios.
Innovation Solution
Adaptive adjustment of aggregation policies based on memory performance parameters, including bandwidth, fault information, and access characteristics to aggregate small IO requests into large requests, optimizing parameters such as aggregation waiting time, threshold, stripe size, and concurrent aggregations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a fixed aggregation policy is used to aggregate IO requests, then aggregation simplicity is maintained, but memory overflow occurs and system performance decreases when bandwidth imbalance or node faults happen
Solution Approach 1:
The patent implements dynamic aggregation policies that automatically adjust aggregation parameters (such as aggregation threshold, waiting time, and stripe size) based on real-time memory pool status, bandwidth conditions, and node health information. This transforms the static fixed policy into a dynamic adaptive system that responds to changing system conditions, resolving the contradiction between operational simplicity and performance reliability.
Solution Approach 2:
The system continuously monitors memory pool capacity, bandwidth usage, and node fault status, using this feedback information to adjust aggregation parameters in real-time. When memory utilization exceeds thresholds or node faults occur, the system automatically modifies aggregation behavior to prevent memory overflow and maintain performance, thereby achieving reliable adaptive operation while keeping the control mechanism transparent.
2Loss of energy
If small IO requests are aggregated into large requests, then network overhead is reduced, but memory may become full quickly when using fixed aggregation parameters
Solution Approach 1:
The patent dynamically adjusts aggregation parameters including aggregation threshold, aggregation waiting time, and aggregation stripe size based on real-time memory pool status. When memory capacity is abundant, larger aggregation thresholds and longer waiting times are used to reduce network overhead by combining more requests. When memory becomes constrained, parameters are automatically reduced to prevent overflow, thus balancing network efficiency with memory utilization.
3Productivity
If high EC ratio is used to improve disk rate, then storage efficiency increases, but performance of small random read/write operations deteriorates
Solution Approach 1:
The patent implements preliminary aggregation of small random read/write requests into larger sequential requests before they reach the storage layer. By aggregating these operations in advance based on memory availability and request patterns, the system reduces the number of EC decoding operations needed, thereby maintaining high disk rates while improving small random operation performance through reduced computational overhead.
4Loss of energy
If aggregation waiting time is extended to aggregate more requests, then network overhead decreases, but memory occupation time increases and performance may be affected
Solution Approach 1:
The system dynamically adjusts aggregation waiting time based on real-time memory pool status and request arrival patterns. When memory capacity is sufficient and request arrival rate is low, longer waiting times are applied to aggregate more requests and reduce network overhead. When memory is constrained or request arrival rate increases, the waiting time is automatically reduced to prevent memory occupation delays, thus adaptively balancing network efficiency with time performance.
Data Source
AI summary
A data processing method is provided. The method includes: obtaining a parameter related to performance of a memory of a storage node, adjusting an aggregation policy based on the parameter, aggregating a first input output IO request in the memory into a second IO request based on an adjusted aggregation policy, and performing persistent storage on data corresponding to the second IO request.