Erasure Code Grouping with Local Parity for Faster Data Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure code technologies in distributed systems lead to high input/output operations per second (IOPS) amplification and significant bandwidth consumption during data recovery due to the need to read multiple data blocks when data loss occurs, especially in multi-AZ environments.
Innovation Solution
An encoding method that divides data blocks into groups, generating local reconstruction parity blocks using other data groups, reducing the number of blocks needed for recovery by utilizing local reconstruction parity blocks stored across different AZs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure code technology is used to store data in a distributed system, then data reliability is ensured with minimal storage overhead, but when data is lost, the system needs to read a sufficient amount of data to recover the original data, leading to IOPS amplification and significant bandwidth consumption
Solution Approach 1:
The patent segments the K data blocks into L data groups (where L > 1), and for each data group, generates a local reconstruction parity block using only the data blocks within that group. This segmentation allows data recovery to be performed locally within each group rather than requiring global data access across the entire distributed system, thereby reducing IOPS amplification and bandwidth consumption while maintaining data reliability through the erasure code mechanism.
2Quantity of substance
If traditional erasure code technology is used to store data in a distributed system, then storage space is optimized, but the number of data blocks that need to be read during data recovery is large, causing high IOPS amplification and bandwidth consumption
Solution Approach 1:
The patent divides data blocks into multiple data groups and generates local reconstruction parity blocks for each group, enabling parallel and independent recovery operations. This segmentation reduces the number of data blocks that need to be read during recovery from the entire dataset to only those within the affected group, significantly reducing data recovery time while maintaining optimized storage space through erasure code.
3Reliability
If traditional erasure code technology is used, then data can be recovered from any k pieces of data out of k+m pieces, but the repair process significantly consumes bandwidth
Solution Approach 1:
The patent segments data into L data groups and generates local reconstruction parity blocks for each group using only the data blocks within that group. This localized approach ensures that when data repair is needed, only the specific data group and its corresponding local reconstruction parity block need to be accessed, dramatically reducing bandwidth consumption compared to traditional erasure code that requires accessing data across the entire distributed system, while maintaining the same fault tolerance capability.
Data Source
AI summary
Encoding methods based on erasure code are disclosed, and a distributed system, a device and a storage medium are provided. The method comprises: acquiring K data blocks of data to be processed; dividing the K data blocks into L data groups, wherein each data group at least comprises K encoding parameters respectively corresponding to the K data blocks, and K and L are integers greater than 1; and for each data group, performing encoding by using at least part of data blocks in other data groups and encoding parameters, which correspond to the at least part of data blocks, in the present data group, to generate a local reconstruction parity block of the present data group. The data groups and the local reconstruction parity blocks are stored in data nodes of a distributed system.