Distributed Erasure Coding with Consolidated Partial Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage systems face challenges in managing large multi-petabyte data capacities and disaster recovery, particularly with geographically distributed erasure coding, which leads to high inter-zone network traffic and resource demands, causing backend zones to become overloaded and vulnerable to data loss.
Innovation Solution
Implementing a two-phase distributed erasure coding process where frontend zones generate partially finished coding chunks and transfer them to backend zones, allowing backend zones to independently complete coding chunks in parallel, reducing inter-zone network traffic and load on backend zones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional geographically distributed erasure coding is implemented, then data protection capability is improved, but inter-zone network traffic increases and backend zones become overloaded
Solution Approach 1:
The erasure coding process is segmented into two distinct phases: a first phase where frontend zones generate partially finished coding chunks from data chunks, and a second phase where backend zones complete the coding chunks using received partial results. This segmentation distributes the computational workload across different zone types, reducing the network traffic burden on backend zones while maintaining data protection capabilities.
Solution Approach 2:
Frontend zones perform preliminary erasure coding operations to generate partially finished coding chunks before transferring them to backend zones. This preliminary action allows backend zones to receive pre-processed data that requires less computational effort to complete, thereby reducing the overall network traffic and computational load on backend zones.
2Reliability
If traditional geographically distributed erasure coding is implemented, then data protection capability is improved, but load on backend zones increases making them vulnerable to data loss
Solution Approach 1:
The system segments the erasure coding workload by assigning the first phase to frontend zones and the second phase to backend zones. This segmentation prevents backend zones from being overloaded with the entire coding process, distributing the computational complexity across the distributed system while maintaining data protection through collaborative completion of coding chunks.
Solution Approach 2:
Partially finished coding chunks act as intermediaries between frontend zones and backend zones. These intermediate results allow frontend zones to contribute to the coding process without transferring complete coding chunks, reducing the computational burden on backend zones while still enabling them to complete the erasure coding operation for data protection.
3Quantity of substance
If consolidated partial coding chunks are used, then inter-zone network traffic is reduced, but coordination between zones is required
Solution Approach 1:
The erasure coding process is divided into two phases with clearly defined responsibilities for frontend and backend zones. This segmentation establishes a coordinated workflow where frontend zones generate and transfer partial results, and backend zones receive and complete the coding chunks, reducing network traffic while maintaining manageable inter-zone coordination through role specialization.
Data Source
AI summary
Processing load and incoming inter-zone network traffic for remote zones during geographically distributed (GEO) erasure coding can be reduced. In one aspect, GEO erasure coding can be performed in multiple phases, wherein each phase can be distributed between zones. Moreover, during a first phase, partially-finished coding chunks can be generated by frontend zones. Further, during an intermediate phase, partially-finished coding chunks that are associated with a common coding chunk can be transferred between the frontend zones and combined to generate consolidated partial coding chunks. Furthermore, during a final phase, the remote zones can receive and combine the consolidated partial coding chunks to generate a complete coding chunk that can be utilized for data recovery.