Distributed Storage Data Rebuild via Local Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed storage systems face inefficiencies in data resynchronization due to redundant data copying, which slows down the process and increases resource consumption, especially during failures when systems are vulnerable and require fast recovery.

Innovation Solution

A two-part deduplication process is implemented, where data pairs with logical addresses and hash values are sent to identify duplicates, allowing for asynchronous marking and later writing, separating the identification and replication tasks to optimize resource usage and reduce data loss risk.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If data redundancy is implemented by copying all data blocks from source to destination node, then data availability is improved, but resynchronization time and resource consumption increase due to redundant copying

Engineering Contradiction:

Improvedata availabilityVSAvoidresynchronization time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent extracts only the essential identification information (hash values and logical addresses) from the data blocks, sending only these metadata elements to the destination node rather than copying the entire data blocks. This allows the system to identify which blocks need replication without transferring redundant data, significantly reducing resynchronization time while maintaining data availability.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes the parameter being transmitted from complete data blocks to condensed metadata (hash values and logical addresses). By transforming the data representation from full blocks to compact identifiers, the system reduces transmission overhead and processing time while still achieving the goal of data redundancy and availability.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If data redundancy is implemented by copying all data blocks from source to destination node, then data availability is improved, but resource consumption increases

Engineering Contradiction:

Improvedata availabilityVSAvoidresource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent performs partial action by sending only the necessary metadata (hash values and logical addresses) rather than complete data blocks. This partial transmission is sufficient to identify duplicate blocks and determine replication needs, avoiding the excessive resource consumption of transferring all data while still achieving the redundancy goal.

Inventive Principle:

Principle #16Partial or excessive action

3Stability of the object's composition

If synchronous data copying is performed during resynchronization, then data consistency is maintained, but system vulnerability increases during the copying process

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem vulnerability

Core Design Contradiction:

Stability of the object's compositionVSReliability

Solution Approach 1:

The patent performs preliminary action by first sending hash values and logical addresses to the destination node before initiating data block copying. This allows the destination node to pre-identify which blocks are duplicates and which need replication, enabling asynchronous processing and reducing the window of vulnerability during resynchronization while maintaining eventual data consistency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the resynchronization process into distinct phases: first transmitting metadata (hash values and logical addresses), then separately copying only the necessary data blocks. This segmentation allows asynchronous processing where identification and replication are decoupled, reducing system vulnerability by avoiding long synchronous operations while ensuring data consistency through the metadata verification step.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11474724B2System and method for speed up data rebuild in a distributed storage system with local deduplication

Publication Date: 2022.10.18 VMWARE INC

AI summary

A method includes obtaining a plurality of representations corresponding respectively to a plurality of blocks of data stored on a source node. A plurality of data pairs are sent to a destination node, where each data pair includes a logical address associated with a block of data from the plurality of blocks of data and the corresponding representation of the block of data. A determination is made whether the blocks of data associated with the respective logical addresses are duplicates of data stored on the destination node. In accordance with an affirmative determination, a reference to a physical address of the block of data stored on the destination node is stored. In accordance with a negative determination, an indication that the data corresponding to the respective logical address is not a duplicate is stored. The data indicated as not being a duplicate is written to the destination node.