Erasure-Coded Incremental Data Reconstruction for OSD Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems using erasure coding for data storage, the existing methods for data reconstruction are inefficient due to the large data volume involved in reconstructing entire data objects, which hampers overall efficiency.
Innovation Solution
A data reconstruction method based on erasure coding that acquires data offset information of incremental data, retrieves corresponding data segments from source OSDs, integrates these segments into an erasure incremental segment, and writes this segment into a to-be-reconstructed OSD, thereby reducing the data volume and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data object is reconstructed from all source OSDs, then data completeness is ensured, but the data volume of reading and writing increases significantly
Solution Approach 1:
The patent segments the data object into multiple data segments distributed across different OSDs. Instead of reconstructing the entire data object, only the specific data segment corresponding to the faulty OSD is reconstructed by retrieving only that segment from source OSDs. This segmentation approach maintains data completeness for the required portion while significantly reducing the data volume involved in the reconstruction process.
2Reliability
If the entire data object is reconstructed, then data restoration is achieved, but the time consumption for data reconstruction increases
Solution Approach 1:
The patent extracts and reconstructs only the specific data segment that corresponds to the faulty OSD, rather than extracting and reconstructing the entire data object. By taking out only the necessary portion (the specific data segment) from the distributed storage system, the reconstruction time is significantly reduced while still achieving complete data restoration for the affected OSD.
3Quantity of substance
If incremental data is identified and reconstructed selectively, then the data volume is reduced, but the complexity of identifying incremental data increases
Solution Approach 1:
The patent performs preliminary actions by maintaining and comparing version information or metadata that tracks which data segments have been updated. Before reconstruction begins, the system identifies which specific data segments are incremental (i.e., need to be reconstructed) by comparing current version information with stored version information. This preliminary identification simplifies the subsequent reconstruction process by clearly delineating which segments require retrieval and reconstruction.
Data Source
AI summary
The present application discloses a data reconstruction method based on erasure coding, an apparatus, a device and a storage medium. The method comprise the following steps: acquiring data offset information of incremental data in a data object; acquiring corresponding data segments from a plurality of source OSDs according to the data offset information; wherein, the source OSDs are target OSDs storing incremental data among respective OSDs storing data objects based on erasure coding, and a quantity of the source OSDs is the same as a quantity of data disks corresponding to the erasure coding; integrating the data segments into an erasure incremental segment, and writing the erasure incremental segment into a to-be-reconstructed OSD, which has no incremental data stored therein, among the respective OSDs. The present method reduces the data volume of data reconstruction, and further ensures the overall efficiency of data reconstruction. In addition, the present application also discloses a data reconstruction apparatus based on erasure coding, a device and a storage medium, with the same beneficial technical effects as above.