Partial Generator Matrix Recovery for Low-I/O Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed systems require high storage redundancy for fault tolerance, leading to increased costs and inefficient I/O operations during data recovery, as they often need to read all valid and check blocks, resulting in high I/O consumption.
Innovation Solution
A method involving a partial generator matrix is used to generate check data for stored data, allowing only partial check and data blocks to be read during recovery, reducing I/O consumption by associating partial check data with specific partial data and irrelevant to other data, thus enabling efficient data recovery without relying on other data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RS method is used to generate check blocks for fault tolerance, then data reliability is improved, but I/O consumption during data recovery increases significantly
Solution Approach 1:
The patent segments the generator matrix into multiple partial generator matrices, where each partial generator matrix corresponds to a specific data block. This segmentation enables the check blocks to be divided into partial check blocks, each associated with specific data blocks. During recovery, only the relevant partial check blocks need to be read, rather than all check blocks, thereby reducing I/O consumption while maintaining data reliability through the distributed redundancy structure.
2Reliability
If multiple copies of data blocks are stored for fault tolerance, then data reliability is improved, but storage cost increases
Solution Approach 1:
The patent transforms the traditional copy-based redundancy approach into an encoding-based redundancy approach using Reed-Solomon codes. Instead of storing multiple identical copies of data blocks, the system encodes data blocks using the generator matrix to produce check blocks with controlled redundancy ratios. By adjusting the parameters of the generator matrix, the system can achieve the desired fault tolerance capability while optimizing storage redundancy to be significantly lower than traditional copying methods.
3Measurement precision
If all valid data blocks and check blocks are read during recovery, then data recovery accuracy is ensured, but I/O consumption increases
Solution Approach 1:
The patent extracts and stores mapping relationships between data blocks and their corresponding partial check blocks. During recovery operations, the system queries this mapping to identify only the specific partial check blocks needed for recovering the lost data block. This extraction approach ensures that the minimum necessary data is read to achieve accurate recovery, eliminating the need to read all valid data blocks and check blocks as required by traditional RS methods.
Data Source
AI summary
The present disclosure provides a method and an apparatus for storing data and a method and an apparatus for recovering data, where the method of recovering data may include: reading partial check data and partial data corresponding to to-be-recovered data from prestored data, where the partial check data is obtained by coding entire data of the to-be-stored data based on a partial generator matrix, the partial generator matrix is obtained by dividing a row of an entire generator matrix and rows of the partial generator matrix which are obtained by dividing include a zero element; and recovering the to-be-recovered data based on the read partial check data and partial data. The examples of the present disclosure can reduce read data volume during data recovery and further reduce I/O consumption substantially.