Erasure-Coded Storage with Adaptive Fragment Ratios for I/O Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems employing erasure coding face input/output (I/O) stress due to uneven distribution of data and coding fragments across nodes, leading to hotspots and performance degradation, especially during scaling-up or scaling-out operations.
Innovation Solution
Implementing a variable ratio of coding fragments to data fragments (VKMR) that deviates from the standard KMR, allowing for proactive I/O load balancing by adjusting the ratio in real-time based on I/O events and storage space conditions, ensuring that new nodes are initially cooled or warmed to match legacy nodes, thereby distributing the load evenly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored using conventional erasure coding with fixed KMR ratio, then data protection is achieved, but I/O stress and hotspots occur due to uneven distribution
Solution Approach 1:
The patent applies dynamics by transitioning from a fixed KMR ratio to a variable VKMR ratio that adapts in real-time based on system conditions. The system dynamically adjusts the ratio of coding fragments to data fragments written to new storage space, allowing the storage system to respond to changing I/O patterns and prevent hotspot formation while maintaining data protection through erasure coding.
Solution Approach 2:
The patent changes the parameter of the KMR ratio to create a variable VKMR ratio. By modifying this critical parameter based on monitored I/O events and storage conditions, the system optimizes the distribution of data and coding fragments across nodes, preventing I/O stress concentration while preserving the mathematical guarantees of erasure coding for data protection.
2Quantity of substance
If new nodes are added to the cluster, then storage capacity increases, but I/O bottlenecks and hotspots are created
Solution Approach 1:
The patent applies preliminary action by proactively adjusting the VKMR ratio before hotspots can form. When new nodes are added to the cluster, the system preemptively modifies the distribution ratio of coding fragments to data fragments, cooling new nodes gradually and preventing I/O bottlenecks before they impact overall system throughput.
3Ease of operation
If coding fragments are distributed evenly across all nodes, then load balancing is achieved, but new nodes cannot be integrated efficiently
Solution Approach 1:
The patent applies local quality by allowing different nodes to have different fragment distribution characteristics. New nodes receive a different ratio of coding fragments compared to legacy nodes, creating localized quality differences that enable efficient node integration while maintaining overall system load balance through the variable VKMR ratio mechanism.
Data Source
AI summary
Adaptive data storing for a data storage system employing erasure coding is disclosed. Incoming data for storage can be encoded according to an erasure code scheme that can generate k coding fragments and in data fragments for each unit of incoming data. The portions of the encoded fragments resulting from the incoming data can be stored among different storage devices of data storage system. A first portion of the encoded fragments can be stored according to a first ratio. A storage system performance indicator based on the storing of the first portion can trigger an update of the first ratio, resulting in a second ratio that can be applied to storage of a second portion of the encoded fragments. The use of the disclosed variable k-to-m ratio can reduce instances of heavily burdened storage devices that can negatively impact performance of the data storage system.