Distributed Erasure Coding for Low-Traffic Chunk Deletion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage architectures face challenges in managing large multi-petabyte data capacities and providing effective data protection across geographically distributed systems, particularly in reducing inter-zone network traffic during data deletion and replication processes.
Innovation Solution
The implementation of a distributed erasure coding system that generates partial coding chunks and directs them across geographical zones to modify complete coding chunks, reducing inter-zone network traffic by performing encoding in two phases and allowing zones to update coding chunks independently without inter-zone communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional replication mechanism is used at chunk level across geographical zones, then data protection is provided, but inter-zone network traffic increases during data deletion and replication processes
Solution Approach 1:
The patent segments the coding chunk into multiple partial coding chunks and distributes them across different zones. Instead of replicating entire coding chunks across all zones, each zone receives and stores only the partial coding chunks it needs, reducing network traffic while maintaining data protection capabilities through distributed erasure coding.
2Reliability
If complete coding chunks are replicated across all geographical zones, then data protection is enhanced, but load balancing between zones deteriorates
Solution Approach 1:
The patent applies local quality by allowing different zones to have different sets of partial coding chunks based on their local conditions and requirements. Each zone processes and stores data locally, and the specific partial coding chunks are distributed according to local needs, enabling load balancing while maintaining data protection through erasure coding.
3Loss of energy
If two-phase encoding is implemented with partial coding chunks, then inter-zone network traffic is reduced, but device complexity increases
Solution Approach 1:
The patent applies preliminary action by performing the first phase of encoding (generating partial coding chunks) at the source zone before distribution to other zones. This preliminary processing reduces the complexity at destination zones, as they only need to perform the second phase (combining partial chunks) rather than complete encoding, thus reducing overall network traffic while managing device complexity.
Data Source
AI summary
Inter-zone network traffic generated during deletion of a data chunk that has been replicated by employing geographically distributed (GEO) erasure coding is reduced. In one aspect, if a data chunk is to be deleted, partial coding chunks are generated by a source zone and provided to destination zones that store complete coding chunks for updating the complete coding chunks based on combining them with the received partial coding chunks. In another aspect, if a first data chunk is to be deleted and a second data chunk is to be replicated, partial coding chunks are generated by the source zone for each data chunk. Further, the partial coding chunks created for different data chunks can be combined to generate transforming chunks, which can then be transferred to the destination zones. The destination zones can then update the complete coding chunks based on combining them with the received transforming chunks.