EC Block Reconstruction Without Decoding in Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face high computation and network resource consumption due to the need for EC decoding when handling faulty nodes, especially with erasure code technologies like Reed-Solomon, which also result in inefficient storage space utilization with multi-copy storage methods.
Innovation Solution
A data reconstruction method where a primary storage node temporarily stores EC blocks on other nodes during faults, allowing for data reconstruction without EC decoding upon node recovery, reducing computation and network resource usage by resending only necessary data to faulty nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If EC decoding is performed to restore data on faulty nodes, then data integrity is maintained, but computation resource consumption increases
Solution Approach 1:
The system performs preliminary actions by having the primary storage node store backup EC blocks and metadata information before the faulty node recovers. When the faulty node comes back online, the data is already prepared and can be directly transferred without performing EC decoding operations, thus maintaining data integrity while avoiding computation resource consumption.
Solution Approach 2:
The primary storage node acts as an intermediary between the erasure code system and the faulty node. Instead of performing EC decoding to restore data, the primary node directly provides the needed EC blocks and metadata to the recovering node, eliminating the need for complex decoding operations and reducing computation resource usage.
2Reliability
If EC decoding is performed for data reconstruction, then lost data is restored, but network bandwidth consumption increases
Solution Approach 1:
The system extracts only the necessary EC blocks and metadata information from the primary storage node and transfers them directly to the faulty node for reconstruction. This selective extraction avoids the need to transfer and process multiple EC blocks required for full EC decoding, thereby restoring lost data while minimizing network bandwidth consumption.
3Reliability
If multi-copy storage technology is used to ensure data security, then data redundancy is achieved, but storage space utilization decreases
Solution Approach 1:
The system changes the parameter of data representation by using erasure codes to encode data into compact EC blocks with associated metadata. Instead of storing multiple full copies of data, the encoded form allows reconstruction of lost data with significantly reduced storage space requirements while maintaining data security and redundancy.
Data Source
AI summary
A data reconstruction method is performed in a distributed storage system. A primary storage node in the distributed storage system performs EC encoding on to-be-written data to generate an EC stripe, and stores, on each storage node, each EC block in the EC stripe. When a fault occurs on some storage nodes and causes a write failure of EC blocks, the primary storage node locally stores an EC block allocated to a storage node to which the write fails and generates metadata information required for data reconstruction. After the storage node recovers from the fault, the primary storage node sends, to the storage node, the stored EC block allocated to the storage node to which the write fails and the metadata information corresponding to the EC block, so that the storage node that recovers from the fault completes the data reconstruction.