Erasure-Coded Network Storage for Low-Overhead Data Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current erasure coding techniques in network storage systems face challenges in reducing network overhead and efficiently verifying data integrity, particularly with Vandermond Reed-Solomon codes, as they require significant computational resources and storage overhead, and existing solutions like Local Repairable Code (LRC) struggle to validate recovered data without incurring excessive network traffic or storage costs.
Innovation Solution
The implementation of a modified erasure coding scheme (k, r, p, m) that divides source data blocks into local and global parity blocks, allowing for efficient data recovery and verification by using local and global parity blocks to reconstruct data from corrupted blocks, reducing network overhead and storage requirements compared to traditional triple replication methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional triple replication is used to protect data integrity, then data reliability is improved, but storage overhead and cost increase prohibitively
Solution Approach 1:
The data is segmented into k source data blocks that are distributed across multiple nodes. Instead of replicating entire data sets three times, the system divides data into manageable blocks and uses erasure coding to generate parity blocks, reducing the total storage overhead while maintaining data integrity through distributed reconstruction capability.
Solution Approach 2:
The system employs a composite protection mechanism combining local parity blocks (p) for quick local recovery and global parity blocks (m) for comprehensive data protection. This layered approach creates a hybrid erasure coding scheme (k, r, p, m) that optimizes both storage efficiency and data reliability by using different types of parity blocks for different recovery scenarios.
2Quantity of substance
If erasure coding schemes are implemented to reduce storage overhead, then storage efficiency is improved, but computational resources and complexity increase
Solution Approach 1:
The erasure coding process is segmented into local and global operations. Local parity blocks are computed from subsets of source data blocks within local groups, reducing the computational scope compared to global erasure coding. This segmentation allows parallel processing and reduces the complexity of individual computation tasks while maintaining overall system efficiency.
Solution Approach 2:
Parity blocks are pre-computed and stored alongside source data blocks during the initial data writing phase. This preliminary action eliminates the need for complex real-time computation during data recovery operations, as the system can directly retrieve and reconstruct data using the pre-existing parity blocks, significantly reducing operational computational complexity.
3Loss of energy
If local repairable code is used to reduce network overhead, then network efficiency is improved, but data verification capability is compromised
Solution Approach 1:
Data blocks are organized into local groups with local parity blocks for efficient local recovery, reducing network traffic for common repair operations. Global parity blocks provide an additional layer that can be used for data verification by reconstructing data through different block combinations, ensuring integrity without requiring excessive network communication for verification purposes.
Solution Approach 2:
The system incorporates verification mechanisms that use global parity blocks to validate recovered data integrity. When data is recovered using local parity blocks, the system can cross-verify the recovered data by reconstructing it through global parity blocks, providing feedback confirmation that the data is correct without requiring additional network traffic to external nodes.
4Reliability
If more parity blocks are added to enhance data protection, then data reliability is improved, but storage requirements and access overhead increase
Solution Approach 1:
Parity blocks are segmented into local parity blocks (p) and global parity blocks (m) with different protection scopes. Local parity blocks protect data within local groups, requiring fewer blocks per group compared to full global protection. This segmentation allows the system to achieve adequate protection levels with fewer total parity blocks, reducing storage requirements while maintaining reliability for common failure scenarios.
Solution Approach 2:
Different levels of protection are applied locally versus globally. Local parity blocks provide quick recovery for single-node failures within a local group, while global parity blocks provide broader protection. This local quality approach ensures that most common failures can be handled by lightweight local parity, reducing the overall number of parity blocks needed compared to uniform global protection schemes.
Data Source
AI summary
Examples include techniques to recover data in a storage. In some examples, an erasure code may be implemented to protect a data file or data object stored to nodes of a networked storage system. Corrupted data included in source data blocks may be recovered and verified based on the erasure code.