Geographically Diverse Erasure Coding for Balanced Fragment Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional erasure coding in geographically diverse data storage systems often fails to efficiently conserve storage space and can lead to increased storage requirements due to the addition of coding fragments, which may result in higher storage needs even compared to replication alone.
Innovation Solution
Implementing a combination of erasure coding and data convolution techniques, where data is split into chunks and convolved across zones, allowing for the deletion of redundant replicas and distribution of coding chunks, thereby reducing storage space while maintaining data accessibility even with the loss of multiple fragments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional erasure coding is implemented in geographically diverse data storage systems, then data redundancy and accessibility are improved, but storage space consumption increases due to the addition of coding fragments
Solution Approach 1:
The patent combines convolution coding with erasure coding to create a hybrid approach. Data is first divided into chunks and convolved within zones, then erasure coding fragments are distributed across geographically diverse zones. This merging of techniques allows for more efficient storage by leveraging both convolution's local redundancy and erasure coding's distributed resilience, reducing total storage space compared to traditional erasure coding alone.
Solution Approach 2:
The patent segments data into chunks that are distributed across multiple geographically diverse zones. Each zone stores portions of the data and coding fragments, allowing the system to tolerate zone failures while maintaining data accessibility. This segmentation enables efficient use of storage space by distributing the redundancy burden across multiple locations rather than concentrating it in one place.
2Reliability
If data is fragmented into multiple pieces and coded according to erasure coding schemes, then data accessibility up to loss of fragments is improved, but the number of fragments and coding fragments increases storage requirements
Solution Approach 1:
The patent divides data into manageable chunks that are then distributed across geographically diverse zones. Each chunk is processed independently through convolution and erasure coding, simplifying the management of individual fragments. This segmentation approach makes the system more tractable by breaking down the complex task of managing large numbers of fragments into smaller, zone-specific operations.
Solution Approach 2:
The patent applies different coding techniques at different levels: convolution coding is applied locally within zones to create initial redundancy, while erasure coding fragments are distributed across zones for broader protection. This local quality approach optimizes each zone's storage characteristics and simplifies fragment management by allowing zone-specific processing rather than requiring global coordination of all fragments.
Data Source
AI summary
Selectively distributing fragments of a data protection set in a geographically diverse data storage system is disclosed. The data protection set can comprise fewer fragments than there are zones comprising the geographically diverse data storage system, which can result in some zones not storing a fragment of the data protection set. Control over distribution of fragments of different data protection sets in the geographically diverse data storage system can mitigate or avoid unbalanced storage of the protection sets. The distribution can be controlled in accordance with a protection set distribution scheme (PSDS). A first PSDS can generate coding fragments from randomly select data fragments of all zones. A second PSDS can generate coding fragments from determined unique zone combinations. A third PSDS can generate coding fragments based on affinity values from an affinity matrix. In embodiments, threshold values or rules can be employed to force generation of a protection set regardless of an applied PSDS where the PSDS excessively retards generation of sufficient protections sets.