Distributed Storage Erasure Coding for Low-Bandwidth Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in optimizing data transfer and read/write operations during system repair, particularly in distributed storage systems, where high bandwidth and disk I/O costs are incurred due to redundancy methods like replication and erasure codes, which do not efficiently manage failures and storage overhead.
Innovation Solution
The Babylon coding method divides data into systematic and parity blocks, using primary and secondary encoding vectors to generate parity functions, reducing the amount of data needed for reconstruction by allowing reconstruction with fewer data blocks, thus minimizing bandwidth and I/O costs while maintaining reliability and storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If replication is used to store data in n different nodes, then fault tolerance is improved (can tolerate n-1 failures), but storage overhead increases by 200% in 3-node replications
Solution Approach 1:
The invention segments data into k data blocks and generates r parity function blocks, storing them in k+r different nodes. This segmentation allows the system to tolerate r failures while using less storage overhead compared to replication, as only r/(k+r) of the total stored data is redundant rather than full copies.
Solution Approach 2:
The invention changes the redundancy parameter from full replication to erasure coding with configurable k and r values. By adjusting these parameters, the system can optimize the balance between fault tolerance (r) and storage efficiency (k/(k+r)), achieving better storage efficiency while maintaining reliability.
2Quantity of substance
If erasure code RS(k,r) is used to break data into k blocks with r parity blocks, then storage efficiency is improved compared to replication, but repair bandwidth and disk I/O costs increase significantly
Solution Approach 1:
The invention pre-structures the erasure code system with specific k and r parameters chosen to optimize repair efficiency. By anticipating failure scenarios, the system configures the code parameters beforehand to minimize the bandwidth and I/O costs during repair operations, rather than optimizing during the actual repair process.
3Reliability
If k providers are needed for data reconstruction in RS(k,r) code, then data reliability is maintained, but the amount of data to be transferred over the network increases to hundreds of terabytes
Solution Approach 1:
The invention extracts only the necessary r parity function blocks from the k+r total blocks for repair operations. Instead of requiring all k data blocks to be transferred for reconstruction, the system can retrieve just the r parity blocks needed to regenerate lost data, significantly reducing network bandwidth consumption during repair.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to a method for coding (k, r) data and to a method for reconstructing data. The coding method comprises steps involving: - dividing an initial data item a into k blocks of data ai; - grouping the k blocks of data into r-1 sub-sets Sj of data blocks; - generating, for each sub-set Sj, a linear function gj(a) defined as a linear combination of the data blocks assigned to said sub-block Sj; and - generating r parity functions comprising a primary parity function f0(a) as a linear combination of the k data blocks ai, and r-1 secondary parity functions, each secondary parity function fj(a) being defined as the sum of the primary parity function f0(a) and of a linear function gj(a).