GEO Erasure Coding Updates for Low-Traffic Chunk Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage architectures face challenges in handling large multi-petabyte data capacities and efficiently managing data protection across geographically distributed storage systems, particularly in reducing inter-zone network traffic and supporting scalable data recovery from dual or multiple zone outages.
Innovation Solution
The implementation of a distributed erasure coding system that generates partial coding chunks within each zone, allowing for localized encoding and decoding operations, thereby reducing inter-zone network traffic and enabling efficient data recovery by replicating and updating coding chunks across zones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional replication mechanism is used at chunk level across geographically distributed zones, then data protection is provided, but inter-zone network traffic increases and load balancing is not achieved
Solution Approach 1:
The patent segments the replication process into two distinct phases: a first phase that generates partial coding chunks locally within the source zone, and a second phase that combines these partial chunks into complete coding chunks at destination zones. This segmentation allows local processing to reduce inter-zone network traffic while maintaining data protection through distributed coding chunks across multiple zones.
Solution Approach 2:
The patent performs preliminary encoding actions at the source zone by generating partial coding chunks before actual data deletion or modification occurs. This preliminary action prepares the distributed storage system in advance, so that when data changes, the corresponding coding chunks can be efficiently updated at destination zones without requiring extensive inter-zone communication during the critical update phase.
2Productivity
If data chunks are deleted in a distributed storage system with geographically distributed zones, then storage capacity is managed, but maintaining coding chunk consistency across zones requires extensive inter-zone communication
Solution Approach 1:
The patent divides the coding chunk update process into two phases: generating partial coding chunks at the source zone (first phase) and combining them at destination zones (second phase). This segmentation simplifies inter-zone communication by reducing the amount of data that needs to be transmitted and processed at each zone, thereby managing storage capacity changes with reduced communication complexity.
Solution Approach 2:
The patent introduces partial coding chunks as intermediaries that are generated at the source zone and then distributed to destination zones. These partial coding chunks serve as mediators that carry only the necessary change information rather than complete coding chunks, reducing the complexity of inter-zone communication during data deletion and capacity management operations.
3Reliability
If geographically distributed erasure coding is implemented, then data recovery from multiple zone outages is enabled, but load balancing between zones is not achieved
Solution Approach 1:
The patent segments the erasure coding process so that the first phase (generating partial coding chunks) occurs locally at the source zone without requiring inter-zone communication, and the second phase (combining partial chunks) occurs at destination zones. This segmentation enables load balancing by distributing the computational workload across zones while maintaining the ability to recover data from multiple zone outages through the distributed coding chunk architecture.
Data Source
AI summary
Inter-zone network traffic generated during deletion of a data chunk that has been replicated by employing geographically distributed (GEO) erasure coding is reduced. In one aspect, if a data chunk is to be deleted, partial coding chunks are generated by a source zone and provided to destination zones that store complete coding chunks for updating the complete coding chunks based on combining them with the received partial coding chunks. In another aspect, if a first data chunk is to be deleted and a second data chunk is to be replicated, partial coding chunks are generated by the source zone for each data chunk. Further, the partial coding chunks created for different data chunks can be combined to generate transforming chunks, which can then be transferred to the destination zones. The destination zones can then update the complete coding chunks based on combining them with the received transforming chunks.