Similarity-Based Deduplication Using DWT Fingerprints and LSH

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional identity-based data deduplication methods are inefficient in finding similar data blocks, especially when memory is limited, and fail to provide real-time deduplication due to the requirement for large in-memory caches, leading to suboptimal compression rates and memory consumption.

Innovation Solution

The implementation of a similarity-based deduplication system using locality-sensitive hashing (LSH) with discrete wavelet transform (DWT) coefficients, where feature-related data is extracted, quantized, and used to construct locality-sensitive fingerprints for efficient similarity detection and differential compression, reducing memory requirements while maintaining high compression rates.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If traditional identity-based deduplication is used to find exact matches, then storage space is saved through hard links, but similar data blocks with modifications are not detected

Engineering Contradiction:

Improvesimilarity detection accuracyVSAvoiddeduplication effectiveness

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent transforms the exact match problem into a similarity measurement problem by changing the parameter from binary equality to continuous similarity degree. This allows detecting modified data blocks by measuring their similarity to original blocks rather than requiring exact matches.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the mechanical exact-match comparison system with a mathematical similarity measurement system using hash functions and distance metrics, enabling detection of semantically similar but byte-different data blocks.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If large in-memory cache is used to provide real-time deduplication, then compression rate improves, but memory consumption increases

Engineering Contradiction:

Improvecompression rateVSAvoidmemory consumption

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent extracts only the essential fingerprint features from data blocks for caching, rather than storing complete blocks. This extraction allows real-time deduplication with minimal memory usage while maintaining high compression rates through efficient similarity detection.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies different quality levels to different parts of the data: full precision for fingerprint calculation, reduced precision for cache storage. This local quality differentiation enables real-time processing with limited memory while preserving deduplication effectiveness.

Inventive Principle:

Principle #3Local quality

3Quantity of substance

If traditional deduplication methods are used with limited memory, then memory requirements are reduced, but real-time deduplication cannot be provided

Engineering Contradiction:

Improvememory consumptionVSAvoidreal-time processing speed

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The patent performs preliminary fingerprint extraction and hashing during data ingestion, preparing compact representations in advance. This preliminary action enables rapid similarity detection during deduplication operations without requiring large memory caches, achieving real-time performance with limited memory.

Inventive Principle:

Principle #10Preliminary action

4Ease of manufacture

If exact match search is used in traditional deduplication, then implementation is simple, but compression rate is suboptimal for modified data

Engineering Contradiction:

Improveimplementation simplicityVSAvoidcompression rate

Core Design Contradiction:

Ease of manufactureVSProductivity

Solution Approach 1:

The patent introduces hash fingerprints as an intermediary between raw data and similarity comparison. This intermediary simplifies implementation by providing fixed-size representations while enabling efficient similarity detection, achieving both simplicity and high compression rates for modified data.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11514666B2Method and system of similarity-based deduplication

Publication Date: 2022.11.29 HUAWEI TECH CO LTD

AI summary

A method of similarity-based deduplication comprising the steps of: receiving an input data block; computing discrete wavelet transform (DWT) coefficients; extracting feature-related DWT data from the computed DWT coefficients; applying quantization to the extracted feature-related DWT data to obtain keys as results of the quantization; constructing a locality-sensitive fingerprint of the input data block; computing a similarity degree between the locality-sensitive fingerprint of the input data block and a locality-sensitive fingerprint of each data block in the plurality of the data blocks in a cache memory; selecting an optimal reference data block as the data block; determining a differential compression is required to be applied based on the similarity degree between the input data block and the optimal reference data block; applying the differential compression to the input data block and the optimal reference data block.