Sub-block deduplication via sector hashing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional deduplication schemes fail to detect redundant data when blocks are similar but not identical, leading to missed opportunities for improving storage efficiency due to the randomizing nature of hash functions.

Innovation Solution

Performing data deduplication at sub-block granularity by selecting a representative sub-block, hashing it, and searching for matches in a deduplication database, allowing for the identification and storage of duplicate ranges within blocks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If conventional block-level deduplication is used, then storage efficiency is improved for identical blocks, but redundant data is missed when blocks are similar but not identical

Engineering Contradiction:

Improvestorage efficiencyVSAvoidduplicate detection accuracy

Core Design Contradiction:

Loss of substanceVSMeasurement precision

Solution Approach 1:

The patent divides a data block into multiple sub-blocks (e.g., 512-byte sectors within a 4KB block) and computes hashes for each sub-block individually. This segmentation allows the system to detect similarities at a finer granularity, identifying duplicate content even when entire blocks differ due to minor variations like timestamps or labels.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of hashing entire blocks, the patent applies hashing to only the representative sub-blocks that are most likely to contain duplicate content. This partial action approach reduces computational overhead while maintaining high detection accuracy for similar data patterns.

Inventive Principle:

Principle #16Partial or excessive action

2Reliability

If hash functions are applied to entire blocks, then unique identification is achieved, but similar blocks with minor differences are incorrectly treated as distinct

Engineering Contradiction:

Improveblock identification accuracyVSAvoidstorage efficiency

Core Design Contradiction:

ReliabilityVSLoss of substance

Solution Approach 1:

The patent applies different treatment to different parts of a block by selecting representative sub-blocks for hashing based on their likelihood to contain duplicate content. This local quality approach focuses computational resources on the most informative portions of data while ignoring minor variations in other regions.

Inventive Principle:

Principle #3Local quality

3Measurement precision

If sub-block hashing is implemented, then duplicate detection accuracy is improved, but computational complexity increases

Engineering Contradiction:

Improveduplicate detection accuracyVSAvoidcomputational complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent computes hashes for only the most representative sub-blocks rather than all possible sub-blocks within a block. This selective partial action maintains high duplicate detection accuracy while significantly reducing the number of hash computations required compared to exhaustive approaches.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11237743B2Sub-block deduplication using sector hashing

Publication Date: 2022.02.01 EMC IP HLDG CO LLC

AI summary

A method of performing deduplication is provided. The method includes (a) selecting, by applying a deterministic selection criterion, a sub-block of a block of data that contains multiple sub-blocks; (b) performing a lookup, into a deduplication table, of a digest generated by hashing the selected sub-block, the lookup matching an entry indexed by the digest in the deduplication table, the entry identifying a previously processed block; and (c) effecting storage of the block, including pointing to the previously processed block. An apparatus, system, and computer program product for performing a similar method are also provided.