Distributed Storage EC Reconstruction Without Decoding Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face high computation and network resource consumption due to the need for EC decoding when handling faulty nodes, especially with erasure code technologies like Reed-Solomon, which require extensive data transfer and decoding operations to restore data.
Innovation Solution
A data reconstruction method that temporarily stores EC blocks on a primary storage node when a fault occurs, allowing for data reconstruction after the faulty node recovers without performing EC decoding, thereby reducing computation and network resource usage by only resending necessary data to the recovered node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If EC decoding is performed to restore data on faulty nodes, then data reliability is maintained, but computation resource consumption increases significantly
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing repair information (parity data and metadata) on the primary storage node before faults occur. When a node recovers, this pre-prepared information is directly transmitted without requiring real-time EC decoding, thus maintaining data reliability while avoiding high computation resource consumption during recovery operations.
Solution Approach 2:
The patent introduces an intermediary mechanism where the primary storage node acts as a mediator that holds and transmits pre-computed repair information to recovering nodes. This intermediary approach replaces the need for complex EC decoding operations between distributed nodes, reducing computation overhead while ensuring data integrity through the use of pre-calculated parity data.
2Reliability
If EC decoding is performed to restore data on faulty nodes, then data completeness is restored, but network bandwidth consumption increases due to extensive data transfer
Solution Approach 1:
The patent extracts only the necessary repair information (specific parity blocks and metadata) from the primary storage node and transmits them directly to recovering nodes, rather than performing full EC decoding that would require extensive data exchange. This extraction approach restores data completeness while minimizing network bandwidth consumption by transmitting only the essential repair data.
3Reliability
If multi-copy storage technology is used to ensure data security, then data reliability is improved, but storage space utilization decreases significantly
Solution Approach 1:
The patent changes the storage redundancy parameter from traditional multi-copy approaches (requiring 2-3 copies) to EC-based redundancy with repair information. This parameter change allows the system to maintain data security and reliability while significantly improving storage space utilization, as EC provides the same reliability with less redundant storage capacity.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
This application relates to the field of IT technologies, and in particular, to a data reconstruction method in a distributed storage system, an apparatus, and a system. A primary storage node in the distributed storage system performs EC encoding on to-be-written data to generate an EC stripe, and stores, on each storage node, each EC block in the EC stripe. When a fault occurs on some storage nodes and causes a write failure of EC blocks, the primary storage node locally stores an EC block allocated to a storage node to which the write fails and generates metadata information required for data reconstruction. After the storage node recovers from the fault, the primary storage node sends, to the storage node, the stored EC block allocated to the storage node to which the write fails and the metadata information corresponding to the EC block, so that the storage node that recovers from the fault completes the data reconstruction. According to the data reconstruction solution in the distributed storage system provided in this application, when some storage nodes are faulty, there is no need to perform EC decoding to restore data on the faulty nodes. Instead, the primary storage node temporarily stores EC blocks allocated to the faulty nodes and sends, after the faulty nodes recover, the temporarily stored EC blocks to the faulty nodes for data reconstruction. According to the foregoing solution, consumption of computation resources that results from EC decoding is avoided when a storage node recovers from a fault and performs data reconstruction, and consumption of network resources that results from a large quantity of data transfer is also avoided when EC decoding is performed.