Hierarchical Erasure Coding for Segment-Level Data Recovery
Solution Overview
Problem
Conventional data recovery methods in geographically diverse storage systems are resource-intensive and inefficient, particularly when dealing with large numbers of compromised chunks, as they often require accessing and processing entire chunks rather than specific segments, leading to high computational burdens and storage overheads.
Innovation Solution
The implementation of hierarchical erasure coding, which combines zone-level and chunk-level erasure coding schemes to encode data into fewer coding chunks, allowing for the recovery of data segments rather than entire chunks, thereby reducing the amount of data that needs to be accessed and processed, and enabling efficient recovery of compromised data without needing to restore entire chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data recovery methods are used to recover compromised chunks in geographically diverse storage systems, then data recovery reliability is achieved, but computational resources and processing overhead increase significantly
Solution Approach 1:
The patent divides data chunks into smaller segments and applies erasure coding at both zone-level and chunk-level. This segmentation allows the system to recover only specific compromised segments rather than entire chunks, significantly reducing the computational resources required for recovery operations while maintaining data reliability.
Solution Approach 2:
The patent implements partial recovery by allowing the system to recover only the necessary portions of compromised data (specific segments) rather than requiring complete chunk recovery. This partial action approach reduces processing overhead while achieving the minimum necessary recovery reliability.
2Reliability
If conventional data recovery methods access entire chunks for segment recovery, then complete data restoration is achieved, but data transfer volume between zones increases
Solution Approach 1:
By segmenting chunks into smaller units and applying hierarchical erasure coding, the patent enables targeted retrieval of only the specific segments needed for recovery. This eliminates the need to transfer entire chunks between zones, significantly reducing data transfer volume while ensuring complete restoration of compromised data.
Solution Approach 2:
The patent extracts and recovers only the specific compromised segments from the distributed storage system rather than retrieving entire chunks. This extraction approach minimizes unnecessary data transfer between geographically distributed zones while achieving complete data restoration.
3Productivity
If hierarchical erasure coding is implemented to enable segment-level recovery, then computational overhead is reduced, but storage system complexity increases
Solution Approach 1:
The patent applies segmentation at multiple levels (zone-level and chunk-level erasure coding) to enable efficient segment recovery. While this multi-level segmentation increases structural complexity, it dramatically improves recovery productivity by allowing targeted recovery operations.
Solution Approach 2:
The patent implements nested erasure coding where zone-level coding contains chunk-level coding. This nested structure, while increasing system complexity, enables hierarchical recovery operations that significantly improve productivity by allowing recovery at different granularities.
4Reliability
If entire chunks are recovered instead of specific segments, then data integrity is ensured, but processing time and resource consumption increase
Solution Approach 1:
The patent segments chunks into recoverable units and uses hierarchical erasure coding to ensure that recovering specific segments maintains data integrity. This segmentation approach reduces processing time by avoiding unnecessary recovery of intact portions of chunks while preserving complete data integrity.
Solution Approach 2:
The patent recovers only the partial amount of data necessary (specific compromised segments) rather than requiring complete chunk recovery. This partial action maintains data integrity for the recovered portions while significantly reducing processing time and resource consumption.
Data Source
AI summary
Recovery of chunk segments stored via hierarchical erasure coding in a geographically diverse data storage system is disclosed. Chunks can be stored according to a first-level erasure coding scheme in zones of a geographically diverse data storage system. The chunks can then be further protected via one or more second-level erasure coding schema within a corresponding zone of the geographically diverse data storage system. In response to determining a segment of a chunk has become less accessible, recovering at least the segment to enable intra-zone recovery of the compromised chunk can be performed according to the hierarchical erasure coding scheme of relevant chunks at relevant zones of the geographically diverse data storage system.



