Zone-Level Erasure Coding for Multi-Zone Data Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data storage architectures struggle to manage and protect large multi-petabyte data capacities, especially in scenarios where all chunk copies become unavailable due to failures or disasters, leading to unrecoverable data losses in geographically distributed storage systems.
Innovation Solution
The implementation of a system that uses non-intersecting sub-matrices of a great coding matrix for erasure coding, allowing for different coding fragments to be generated and stored across zones, enabling enhanced data recovery even when all chunk copies are unavailable by leveraging the combination of erasure coding and replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional replication mechanism is used at chunk level across zones, then data protection is provided, but data remains unrecoverable when all chunk copies become unavailable due to failures or disasters
Solution Approach 1:
The coding matrix is segmented into multiple non-intersecting sub-matrices, where each sub-matrix is stored in a different zone. This segmentation allows any single zone to provide complete coding information for data recovery, eliminating the unrecoverable data loss problem while distributing complexity across zones rather than increasing overall system complexity.
Solution Approach 2:
The solution transitions from traditional replication (storing identical copies) to erasure coding with distributed matrix segments across spatial zones. This dimensional redistribution of coding information across multiple zones enables recovery from complete zone failures while maintaining manageable complexity through mathematical distribution rather than physical replication.
2Reliability
If erasure coding is implemented across geographically distributed zones, then data protection is improved, but additional capacity overhead is required
Solution Approach 1:
Instead of implementing complete erasure coding across all zones (which would require significant capacity overhead), the patent applies partial erasure coding by storing only the essential coding matrix sub-matrices in each zone. This partial approach provides sufficient data protection for zone failures without the full capacity overhead of complete distributed erasure coding across all storage locations.
3Reliability
If non-intersecting sub-matrices are used for erasure coding across zones, then data recovery is enabled when all chunk copies are unavailable, but coding and decoding complexity increases
Solution Approach 1:
The coding matrix is divided into non-intersecting sub-matrices distributed across zones, with each sub-matrix being independently processable. This segmentation reduces the complexity of coding and decoding operations at each zone compared to handling the complete coding matrix centrally, while still enabling full data recovery through the combination of distributed sub-matrices.
4Ease of operation
If traditional chunk-level replication is used, then simple storage management is maintained, but data availability decreases when disasters affect all chunk copies
Solution Approach 1:
The coding matrix segmentation into zone-distributed sub-matrices maintains relatively simple storage management at each zone (storing local sub-matrices and data fragments) while dramatically improving data availability. When disasters affect traditional replicated chunks, the segmented matrix approach ensures that coding information remains accessible across zones, enabling data reconstruction without the complete unavailability that plagues traditional replication.
Data Source
AI summary
In geographically-distributed object storage systems that utilize erasure coding, non-intersecting sub matrices of a great encoding matrix can be utilized to erasure code data fragments of a chunk at the zone level and generate coding fragments. Accordingly, the data fragments that are stored within different zones are identical, while the coding fragments stored within the different zones are disparate. Subsequent to a multi-zone data failure, wherein it is determined that a decoding operation cannot be performed at the zone level, available fragments associated with the chunk can be collected from the zones and collectively decoded to recover the chunk. In one aspect, the fragments can be decoded by utilizing a great decoding matrix that corresponds to the great encoding matrix.