Deduplication Appliance Remote Restore Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for restoring segmented, deduplicated files from remote storage tiers either incur high latency and increased local storage costs or result in unnecessary bandwidth usage and higher customer bills due to inefficient data retrieval strategies.

Innovation Solution

Implementing a method that prefetches file segments from both local and remote storage tiers based on their location, using fingerprints and metadata to optimize data retrieval, thereby reducing latency and costs while ensuring efficient data access.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of energy

If files are recalled from remote tier to local tier before restore, then bandwidth efficiency is improved, but local storage space consumption increases and latency is worsened

Engineering Contradiction:

Improvebandwidth efficiencyVSAvoidlocal storage space consumption

Core Design Contradiction:

Loss of energyVSQuantity of substance

Solution Approach 1:

The patent segments the file restore process into two independent paths: (1) recalling only missing segments from remote tier to local tier, and (2) directly reading available segments from local tier without recall. This segmentation allows the system to avoid recalling entire files when only portions are needed, thereby reducing local storage space consumption while maintaining bandwidth efficiency for the necessary data transfer.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by recalling only the necessary portions of files from remote tier to local tier based on what is actually needed for the restore operation, rather than recalling entire files. The system determines which segments are already available locally and only recalls the missing segments, thus avoiding excessive local storage space consumption while maintaining bandwidth efficiency for the essential data transfer.

Inventive Principle:

Principle #16Partial or excessive action

2Quantity of substance

If files are directly restored from remote tier, then local storage space consumption is reduced, but latency increases and bandwidth efficiency worsens

Engineering Contradiction:

Improvelocal storage space consumptionVSAvoidlatency

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent segments the file restore process into two independent paths: (1) recalling only missing segments from remote tier to local tier, and (2) directly reading available segments from local tier without recall. This segmentation allows the system to avoid the full latency of remote tier access for segments already available locally, thereby reducing overall restore latency while maintaining low local storage space consumption.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-fetching segments from the remote tier to the local tier before the actual restore operation begins, when possible. This preliminary action reduces the latency impact during the actual restore by having data ready in advance, while still maintaining the benefit of not permanently storing recalled files on the local tier.

Inventive Principle:

Principle #10Preliminary action

3Ease of operation

If all segments are fetched from remote tier during direct restore, then data access simplicity is improved, but bandwidth usage increases and costs increase

Engineering Contradiction:

Improvedata access simplicityVSAvoidbandwidth usage

Core Design Contradiction:

Ease of operationVSLoss of energy

Solution Approach 1:

The patent extracts and removes the unnecessary step of fetching all segments from the remote tier during direct restore. Instead, it extracts only the missing segments that are not already available on the local tier, thereby reducing bandwidth usage and costs while maintaining the simplicity of the direct restore approach for the remaining data.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies partial action by fetching only the necessary portions of data from the remote tier during direct restore operations, rather than fetching all segments. The system identifies which segments are already available locally and only fetches the missing segments, thus reducing bandwidth usage and costs while maintaining data access simplicity for the complete file restore.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS10831391B2Method to serve restores from remote high-latency tiers by reading available data from a local low-latency tier in a deduplication appliance

Publication Date: 2020.11.10 EMC IP HLDG CO LLC

AI summary

Embodiments are described for performing file restores from remote high-latency storage tiers by reading available data from a local low-latency tier in a deduplication appliance. A request to restore a previously segmented and deduplicated file can be received by a storage appliance from an application, each segment having a fingerprint. The name of the file can be looked up in an index on the storage appliance, and a first batch of fingerprints of segments of the file can be retrieved from the index. Each fingerprint can be looked up in metadata in the index to determine whether the segment corresponding to the fingerprint is available locally and therefore need not be retrieved locally. A list of local and remote prefetch segments is generated, and a prefetch request is generated for each list, if non-empty. Use of the prefetch scheme can be dynamically turned on or off.