Incremental Erasure Coding for Lower Storage Write Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure code technologies require full encoding of data blocks to k before generating parity blocks, leading to inefficient use of storage space and high write overheads, especially in scenarios where data is written incrementally.
Innovation Solution
An erasure code-based encoding method that allows incremental encoding of data blocks, generating parity blocks on the fly and updating global parity blocks through exclusive OR operations, optimizing encoding flexibility and reducing storage overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full erasure code encoding is performed only when data blocks reach k, then data reliability is ensured, but storage space is wasted and write overhead increases
Solution Approach 1:
The patent segments the encoding process into incremental steps rather than requiring complete accumulation of k data blocks. Each incremental encoding operation processes a subset of data blocks independently, allowing parity blocks to be generated progressively as data arrives, thus improving storage space utilization while maintaining reliability guarantees
Solution Approach 2:
The patent performs preliminary encoding operations on subsets of data blocks before all k blocks are available. By pre-computing parity information incrementally, the system prepares data for future reconstruction needs without waiting for complete data accumulation, reducing write overhead and improving storage efficiency
2Reliability
If full erasure code encoding is performed only when data blocks reach k, then encoding correctness is maintained, but encoding efficiency decreases
Solution Approach 1:
The patent implements continuous incremental encoding operations that process data blocks as they arrive rather than waiting for batch completion. This continuous processing approach maintains encoding correctness through systematic parity generation while significantly improving encoding efficiency by eliminating idle waiting time and reducing computational hotspots
Solution Approach 2:
The patent introduces dynamic encoding that adapts to the current state of data availability. The encoding process transitions from static batch processing to dynamic incremental processing, adjusting computation intensity and timing based on data arrival patterns, thereby maintaining correctness while optimizing encoding throughput
3Reliability
If three-copy redundancy protection is used when data blocks are less than k, then data reliability is maintained, but storage overhead increases
Solution Approach 1:
The patent dynamically changes the redundancy parameter from fixed three-copy protection to variable incremental parity based on the current number of available data blocks. This parameter adaptation allows the system to use minimal necessary redundancy at each encoding stage, reducing storage overhead while maintaining reliability through mathematically guaranteed reconstruction capability
Data Source
AI summary
A method includes an encoder side that obtains a plurality of data blocks in batches, and encodes the data blocks. In an ith time of encoding, after obtaining an incremental data block, the encoder side performs encoding processing on the incremental data block using erasure code to generate an incremental parity block, the incremental data block includes at least one data block, and i is an integer greater than 1. The encoder side performs an exclusive OR operation based on the incremental parity block and an original global parity block to generate a first incremental global parity block. The original global parity block is a first incremental global parity block generated by the encoder side in an (i−1)th time of encoding, and an original data block is a data block obtained by the encoder side before the (i−1)th time of encoding.