Distributed Storage Erasure Coding Write Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data redundancy methods in storage systems, such as RAID and erasure coding, incur increased read and write overheads, and existing solutions like BLRC do not fully optimize write efficiency in distributed storage systems.
Innovation Solution
The proposed solution involves a distributed storage system that employs erasure coding within and across storage nodes, with synchronous and asynchronous storage of intra-node and inter-node parity chunks to minimize write amplification and optimize write efficiency, using a 2+1 erasure coding scheme within nodes and 3+2 erasure coding across nodes, and mirroring for enhanced fault tolerance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data redundancy is implemented using traditional RAID or erasure coding methods, then fault tolerance is improved, but write overhead and write amplification increase
Solution Approach 1:
The patent segments parity computation into two distinct parts: intra-node parity (computed within each storage node) and inter-node parity (computed across storage nodes). This segmentation allows different storage nodes to independently compute their respective parities without waiting for other nodes, thereby reducing write overhead while maintaining fault tolerance through both levels of redundancy.
Solution Approach 2:
The patent implements preliminary action by having each storage node pre-compute its intra-node parity chunks locally before receiving data from other nodes. This preliminary computation eliminates the need for nodes to wait for data from others during the write operation, significantly reducing write amplification and improving write efficiency while preserving fault tolerance capabilities.
2Quantity of substance
If BLRC erasure coding scheme is used with flexible balance between storage overhead and cross-zone network traffic, then storage efficiency is improved, but write amplification remains high in distributed storage systems
Solution Approach 1:
The patent applies local quality by having each storage node independently compute intra-node parity chunks using local data chunks and local inter-node parity chunks, rather than requiring global computation across all nodes. This localized approach reduces network traffic and write amplification while maintaining storage efficiency through the same erasure coding schemes (2+1 within nodes, 3+2 across nodes).
Data Source
Figure 1
Figure 2
Figure 3
AI summary
According to examples, a storage node may include storage devices and a controller that may determine whether all of a plurality of data chunks of a first intra-node portion of a stripe have been stored on the storage node. Based on a determination that all of the data chunks have been stored, a first intra-node parity chunk may be stored at a second one of the storage devices, in which the first intra-node parity chunk may be determined from at least one of the data chunks of the first intra-node portion. Based on a determination that at least one of the data chunks has not been stored, storage of a first intra-node parity chunk of the stripe on the storage node may be delayed until a determination is made that all of the data chunks of the first intra-node portion have been stored at the storage node.