Distributed Storage Replica Recovery via Erasure Codes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current data replication systems face challenges in balancing safety and efficiency, as maintaining multiple replicas increases storage costs, network traffic, and energy consumption, while reducing replicas heightens the risk of data loss due to disk failures.

Innovation Solution

A distributed storage system that uses a pair of replicas per data unit, with each replica stored in contiguous segments across different drives, and employs erasure codes to facilitate recovery even if one replica is lost, allowing for efficient reconstruction of failed storage drives with reduced risk of data loss.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multiple replicas are stored for each data unit, then data safety and fault tolerance are improved, but storage efficiency deteriorates due to reduced usable space, increased network traffic, and higher energy consumption

Engineering Contradiction:

Improvedata safetyVSAvoidstorage efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system dynamically adjusts the number of replicas based on data access patterns and failure history. Frequently accessed or critical data maintains multiple replicas, while less critical data uses fewer replicas, optimizing the balance between safety and storage efficiency

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The replication factor is not fixed but adapts over time based on system conditions, data importance, and observed failure rates. The system can increase or decrease replica counts dynamically to respond to changing requirements

Inventive Principle:

Principle #15Dynamics

2Reliability

If additional replicas are maintained, then fault tolerance is improved, but network traffic increases due to synchronization requirements

Engineering Contradiction:

Improvefault toleranceVSAvoidnetwork traffic

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system pre-synchronizes replicas before failures occur by maintaining update logs and change trackers. When a failure is detected, the recovery process uses these pre-prepared synchronization markers to resume replication from the last known good state, avoiding re-transmission of unchanged data

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

During normal operation, the system skips unnecessary synchronization transmissions by comparing data checksums and version markers between replicas, only transferring data that has actually changed

Inventive Principle:

Principle #21Skipping (Rushing through)

3Reliability

If more replicas are stored, then data safety is improved, but energy consumption increases due to additional disk and network activity

Engineering Contradiction:

Improvedata safetyVSAvoidenergy consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system places replicas on storage devices with different energy characteristics and operational states. Critical replicas are maintained on always-on devices, while less critical replicas may reside on devices that can enter low-power states, optimizing the energy-safety tradeoff locally for each replica

Inventive Principle:

Principle #3Local quality

4Productivity

If fewer replicas are maintained, then storage efficiency is improved, but the risk of data loss increases due to simultaneous disk failures

Engineering Contradiction:

Improvestorage efficiencyVSAvoiddata loss risk

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system divides data into segments and distributes replicas across different failure domains (different physical disks, different storage arrays, different locations). This segmentation ensures that a single failure event cannot affect all replicas, maintaining safety even with fewer total replicas

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system introduces an intermediary coordination service that monitors storage device health and orchestrates replica placement and recovery. This intermediary can detect failures early and initiate recovery procedures before data is lost, enabling the system to operate safely with minimal replicas

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS9535802B2Management and recovery of distributed storage of replicas

Publication Date: 2017.01.03 TECHNION RES & DEV FOUND LTD

AI summary

A method of data replica recovery that is based on separate storage drives connected to a network where each storage drive has a storage space divided to contiguous storage segments and is electronically connected to a memory support component via a connection. Pairs of replicas, each of one of a plurality of data units, are stored in a manner that allows, in response to detection of a storage failure in one storage drive, to create replacement replicas in the memory support components of the other storage drives to assure that two replicas of each data unit can be found in the storage system.