Asynchronous Semi-Inline Deduplication for Storage Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing deduplication techniques, such as post-process and inline deduplication, face challenges in storage efficiency and latency, with post-process deduplication causing on-disk fragmentation and resource-intensive processing, while inline deduplication impacts write request latency and requires additional storage resources.

Innovation Solution

Asynchronous semi-inline deduplication is performed at a first storage tier, such as solid state storage, before data is moved to a second tier, reducing latency and fragmentation by eliminating redundant data and minimizing resource usage, thereby reducing data transfer bandwidth and storage allocation needs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If post-process deduplication is performed, then storage efficiency is improved, but on-disk fragmentation occurs and additional system resources are consumed

Engineering Contradiction:
Improveredundant dataVSAvoidon-disk fragmentation
Core Design Contradiction:
Loss of substanceVSObject-generated harmful factors

Solution Approach 1:

The patent performs deduplication operations before data is fully written to disk and before I/O completion is returned to the client. By evaluating and eliminating redundant data in advance during the write path, the system prevents on-disk fragmentation from occurring in the first place, rather than dealing with it after the fact.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary deduplication evaluation process that operates between data receipt and disk write completion. This intermediary layer identifies and removes redundant data blocks before they are permanently written to disk, using a mechanism that doesn't directly block the write path but still achieves deduplication.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Loss of substance

If post-process deduplication is performed, then storage efficiency is improved, but additional system resources are consumed for sorting and hashing

Engineering Contradiction:
Improveredundant dataVSAvoidsystem resources
Core Design Contradiction:
Loss of substanceVSUse of energy by moving object

Solution Approach 1:

The patent performs deduplication evaluation during the write path before data is committed to disk, rather than performing resource-intensive sorting and hashing operations after data is already stored. This preliminary action reduces the need for extensive post-processing resource consumption.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent replaces traditional mechanical deduplication approaches (sorting and hashing tables) with a fingerprint-based comparison mechanism. By using fixed-size fingerprints and direct comparison algorithms, the system reduces computational overhead and resource consumption associated with maintaining large hashing tables and sorting operations.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

3Loss of substance

If inline deduplication is performed, then redundant data is eliminated before reaching storage, but write request latency increases

Engineering Contradiction:
Improveredundant dataVSAvoidwrite request latency
Core Design Contradiction:
Loss of substanceVSLoss of time

Solution Approach 1:

The patent introduces an intermediary deduplication evaluation process that operates between data receipt and disk write completion. This intermediary layer identifies and removes redundant data blocks before they are permanently written to disk, using a mechanism that doesn't directly block the write path but still achieves deduplication.

Inventive Principle:
Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent performs deduplication evaluation during the write path before data is committed to disk, rather than performing resource-intensive sorting and hashing operations after data is already stored. This preliminary action reduces the need for extensive post-processing resource consumption.

Inventive Principle:
Principle #10Preliminary action

4Loss of substance

If post-process deduplication is performed, then redundant data is removed, but I/O performance degrades during deduplication processing

Engineering Contradiction:
Improveredundant dataVSAvoidI/O performance
Core Design Contradiction:
Loss of substanceVSProductivity

Solution Approach 1:

The patent performs deduplication evaluation during the write path before data is committed to disk, rather than performing resource-intensive sorting and hashing operations after data is already stored. This preliminary action reduces the need for extensive post-processing resource consumption.

Inventive Principle:
Principle #10Preliminary action

Data Source

PatentUS11620064B2Asynchronous semi-inline deduplication
Publication Date: 2023.04.04 NETAPP INC
  • US11620064B2 patent drawing
  • US11620064B2 patent drawing
  • US11620064B2 patent drawing

AI summary

Techniques are provided for asynchronous semi-inline deduplication. A multi-tiered storage arrangement comprises a first storage tier, a second storage tier, etc. An in-memory change log of data recently written to the first storage tier is evaluate to identify a fingerprint of a data block recently written to the first storage tier. A donor data store, comprising fingerprints of data blocks already stored within the first storage tier, is queried using the fingerprint. If the fingerprint is found, then deduplication is performed for the data block to create deduplicated data based upon a potential donor data block within the first storage tier. The deduplicated data is moved from the first storage tier to the second storage tier, such as in response to a determination that the deduplicated data has not been recently accessed. The deduplication is performed before cold data is moved from first storage tier to second storage tier.