Erasure Coding Parity Merging for Low-Data-Movement Storage Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional distributed storage systems face challenges with network bandwidth limitations, leading to increased latency and inefficient resource utilization due to suboptimal data distribution and high write amplification, despite sufficient storage capacity.
Innovation Solution
The system employs a co-processor-based erasure coding method that generates partial EC codewords with intermediate parity fragments, distributing and merging them among storage nodes to reduce data movement and network traffic, thereby improving resource efficiency and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional distributed storage systems store multiple copies of data in multiple storage nodes, then data reliability is improved, but write amplification increases and storage capacity utilization decreases
Solution Approach 1:
The patent segments data into data fragments and generates separate parity fragments through erasure coding. Instead of storing complete data copies, the system divides data into k fragments and generates n-k parity fragments, storing all n fragments across different storage nodes. This segmentation approach maintains data reliability while reducing write amplification compared to traditional replication.
Solution Approach 2:
The patent changes the redundancy parameter from full replication (storing complete copies) to erasure coding (storing fragmented data and parity). By using erasure coding with configurable redundancy ratios (e.g., k:n where n>k), the system achieves the same reliability level with reduced storage overhead and lower write amplification compared to traditional multi-copy approaches.
2Reliability
If traditional distributed storage systems maintain multiple copies of data across storage nodes, then data availability is improved, but network load and data movement increase
Solution Approach 1:
The system segments data and parity into separate fragments distributed across storage nodes. When data needs to be written or recovered, only the specific data fragments and corresponding parity fragments need to be accessed, rather than moving entire data copies. This segmentation reduces network load and data movement compared to traditional replication approaches.
Solution Approach 2:
The patent extracts the parity generation function from the main data path and performs it locally at storage nodes. By generating parity fragments locally using erasure coding algorithms rather than transmitting complete data copies across the network, the system significantly reduces network load while maintaining data availability.
3Reliability
If distributed storage systems use traditional erasure coding with full data movement, then data protection is improved, but latency increases due to extensive data movement
Solution Approach 1:
The patent performs preliminary erasure coding operations at the storage node before data is fully received or during incremental writes. By generating intermediate parity fragments locally and storing them immediately, the system avoids waiting for complete data sets to be assembled and moved across the network, thereby reducing latency while maintaining data protection.
Solution Approach 2:
The system extracts and performs erasure coding operations locally at storage nodes rather than centralizing them. This local execution of parity generation eliminates network traversal delays for parity data, significantly reducing latency compared to traditional approaches where all data movement must complete before parity can be generated and stored.
4Quantity of substance
If distributed storage systems increase storage capacity, then storage capability is improved, but network bandwidth becomes a bottleneck
Solution Approach 1:
The patent changes the data distribution parameter from replication to erasure coding, allowing the system to scale storage capacity without proportionally increasing network bandwidth requirements. With erasure coding, adding storage nodes increases capacity while parity fragments are generated and stored locally, avoiding the need to transmit complete data copies across the network.
Solution Approach 2:
By segmenting data into fragments and generating corresponding parity fragments, the system allows storage capacity to be increased by adding more storage nodes without requiring proportional increases in network bandwidth. Each node stores only its assigned fragments, and parity generation happens locally, decoupling storage capacity scaling from network bandwidth requirements.
Data Source
AI summary
A system is provided for performing erasure coding (EC) in a distributed storage system. During operation, the system can perform a partial encoding of a received first set of data fragments and second set of data fragments using EC to generate a first and a second EC codeword, respectively. The system can then distribute the first and the second set of data fragments among a set of storage nodes within the distributed storage system. The system can also distribute a first and the second set of intermediate parity fragments in the first and second EC codeword, respectively, among a subset of the storage nodes with alignments. The system can then merge the first and the second set of intermediate parity fragments to generate an overall parity for both the first and the second set of data fragments. The system can store, based on the alignments, each overall parity fragment in the overall parity in the corresponding subset of storage nodes.