Distributed Storage Erasure Coding for Low-Bandwidth Repair
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in optimizing the bandwidth and disc input/output costs during data repair after failures, while maintaining the same storage efficiency and reliability as conventional erasure codes.
Innovation Solution
A data coding method that divides initial data into k blocks, groups them into r-1 subsets, generates primary and secondary parity functions, and uses encoding vectors to reduce the data blocks needed for repair, allowing for efficient reconstruction of lost data blocks with reduced bandwidth and I/O costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional Reed-Solomon erasure codes are used for data redundancy, then reliability and storage efficiency are maintained, but repair bandwidth and disc input/output costs increase significantly
Solution Approach 1:
The patent segments the k data blocks into multiple groups (e.g., k1, k2, ..., km where sum of ki equals k). For each group, separate parity blocks are generated using different generator polynomials. This segmentation allows repair operations to access only the relevant subset of data blocks and parity blocks needed for the specific failed node, rather than requiring access to all k data blocks as in conventional RS codes, thereby reducing repair bandwidth and I/O costs while maintaining the same reliability guarantees
Solution Approach 2:
The patent introduces an additional dimension of organization by creating multiple parity blocks for each data block group using different generator polynomials (g1(x), g2(x), ..., gm(x)). This multi-dimensional parity structure enables selective access to specific parity blocks during repair operations, reducing the amount of data that needs to be transferred over the network and read from discs, thus addressing the bandwidth and I/O cost problem while preserving reliability
2Reliability
If Reed-Solomon erasure codes are used for data redundancy, then reliability is maintained, but disc input/output costs during repair increase
Solution Approach 1:
By segmenting data blocks into groups and generating separate parity blocks for each group, the patent enables faster repair operations. When a node fails, the system only needs to read the specific subset of data blocks and corresponding parity blocks for that group, rather than reading all k data blocks from disc. This selective access significantly reduces disc I/O costs and accelerates the repair process while maintaining the same reliability through the distributed parity structure
3Ease of operation
If replication is used instead of erasure codes, then repair operations are simpler, but storage overhead increases to 200%
Solution Approach 1:
The patent merges the advantages of replication simplicity with the storage efficiency of erasure codes. By organizing data into groups and creating targeted parity blocks for each group using different generator polynomials, the system achieves repair simplicity similar to replication (where only specific blocks need to be accessed) while maintaining the 40% storage overhead efficiency of erasure codes rather than the 200% overhead of full replication
Data Source
AI summary
A method for coding (k, r) data and a method for reconstructing data are provided. The coding method includes steps consisting in: dividing an initial datum a into k data blocks ai; grouping the k data blocks into r−1 subsets Sj of data blocks; generating, for each subset Sj, a linear function gj(a) defined as a linear combination of the data blocks assigned to said subset Sj; and generating r parity functions comprising a primary parity function f0(a) as a linear combination of the k data blocks ai, and r−1 secondary parity functions, each secondary parity function fj(a) being defined as the sum of the primary parity function f0(a) and of a linear function gj(a).