Distributed Erasure Coding for Low-Traffic Disaster Recovery

Solution Overview

Problem

Traditional storage architectures face challenges in managing large multi-petabyte data capacities and providing efficient disaster recovery, especially in geographically distributed systems, where inter-zone network traffic and resource demands can lead to backend zone overload and inefficient data protection.

Innovation Solution

The implementation of distributed erasure coding across geographical zones, where data chunks are encoded in two phases and distributed between zones to reduce inter-zone network traffic and backend zone load, allowing for efficient data recovery and protection without relying on traditional mirroring or parity protection methods.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional replication mechanism is used for geographically distributed setups, then data protection is provided, but inter-zone network traffic increases and backend zones become overloaded

Engineering Contradiction:

Improvedata protectionVSAvoidinter-zone network traffic

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The erasure coding operation is segmented into two distinct phases: a first phase that generates partial coding chunks locally at each zone, and a second phase that combines these partial chunks into final coding chunks. This segmentation allows data protection to be achieved with reduced inter-zone network traffic, as only partial coding chunks need to be transmitted rather than complete coding chunks or full data replicas.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The first phase of erasure coding is performed preliminarily at each zone before data is sent to backend zones. Each zone generates partial coding chunks from its data chunks in advance, so that when data needs to be protected or recovered, the heavy lifting of initial encoding has already been done locally, reducing the burden on backend zones and minimizing inter-zone network traffic.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If traditional replication mechanism is used for geographically distributed setups, then data protection is provided, but backend zone load increases

Engineering Contradiction:

Improvedata protectionVSAvoidbackend zone load

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The erasure coding operation is segmented into two distinct phases: a first phase that generates partial coding chunks locally at each zone, and a second phase that combines these partial chunks into final coding chunks. This segmentation allows data protection to be achieved with reduced inter-zone network traffic, as only partial coding chunks need to be transmitted rather than complete coding chunks or full data replicas.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The first phase of erasure coding is performed preliminarily at each zone before data is sent to backend zones. Each zone generates partial coding chunks from its data chunks in advance, so that when data needs to be protected or recovered, the heavy lifting of initial encoding has already been done locally, reducing the burden on backend zones and minimizing inter-zone network traffic.

Inventive Principle:

Principle #10Preliminary action

3Loss of energy

If distributed erasure coding is implemented, then inter-zone network traffic is reduced, but system complexity increases

Engineering Contradiction:

Improveinter-zone network trafficVSAvoidsystem complexity

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The erasure coding operation is segmented into two distinct phases: a first phase that generates partial coding chunks locally at each zone, and a second phase that combines these partial chunks into final coding chunks. This segmentation allows data protection to be achieved with reduced inter-zone network traffic, as only partial coding chunks need to be transmitted rather than complete coding chunks or full data replicas.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system implements a universal two-phase erasure coding mechanism that can be applied across all zones in the geographically distributed storage system. The same encoding logic and chunk combination process is used regardless of which zone is involved, providing a multi-functional solution that handles data protection, recovery, and traffic reduction in a unified manner.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10572191B1Disaster recovery with distributed erasure coding

Publication Date: 2020.02.25 EMC IP HLDG CO LLC

AI summary

Processing load for remote zones and inter-zone network traffic during geographically distributed (GEO) erasure coding can be reduced. In one aspect, GEO erasure coding can be performed in two phases, wherein each phase can be distributed between zones. Moreover, during a first phase, partially-finished coding chunks can be generated and provided to the remote zones. Further, during a second phase the remote zones can combine the received partially-finished coding chunks independently, to generate a complete coding chunk that can be utilized for data recovery.