Similarity-Based Deduplication Using DWT Fingerprints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional identity-based data deduplication methods are inefficient in detecting similar data blocks, especially when memory is limited, and fail to provide real-time deduplication due to the need for large in-memory caches, leading to suboptimal compression rates and increased memory consumption.
Innovation Solution
The implementation of a similarity-based deduplication system using locality-sensitive hashing (LSH) with discrete wavelet transform (DWT) coefficients, where feature-related data is extracted, quantized, and used to construct locality-sensitive fingerprints for high-speed similarity detection and differential compression, reducing memory requirements while maintaining effective compression rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional identity-based data deduplication is used to achieve exact matches, then storage space is saved, but similar data blocks with modifications are not detected
Solution Approach 1:
The patent transforms the exact match requirement into a similarity measurement problem by changing the parameter from binary equality to continuous similarity degree. This allows detecting modified data blocks while maintaining compression efficiency through threshold-based decision making.
Solution Approach 2:
The patent replaces the mechanical exact-match comparison system with a mathematical similarity measurement system using dot product and cosine similarity calculations, enabling detection of semantically similar blocks even with modifications.
2Speed
If large in-memory cache is used to provide real-time deduplication, then similarity detection speed is improved, but memory consumption increases
Solution Approach 1:
The patent segments the large cache into multiple smaller partitions, each handled by separate worker threads. This allows parallel processing of data blocks across partitions, achieving real-time performance without requiring a single large memory space.
Solution Approach 2:
The patent implements dynamic load balancing and adaptive partitioning where worker threads can dynamically access different partitions based on workload, optimizing memory utilization while maintaining real-time processing capability.
3Quantity of substance
If traditional deduplication with hard links is used for small blocks, then storage space is optimized, but similarity-based compression is not achieved
Solution Approach 1:
The patent changes the approach from binary exact-match parameter to continuous similarity parameter, enabling the system to handle modified data blocks by measuring their similarity degree and applying appropriate compression strategies.
Solution Approach 2:
The patent applies partial compression by storing only the difference between similar blocks rather than complete duplicates, achieving space savings for modified blocks while maintaining the ability to reconstruct original data when needed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method of similarity-based deduplication is disclosed, comprising the steps of: receiving an input data block; computing discrete wavelet transform, DWT, coefficients based on the input data block; extracting feature-related DWT data from the computed DWT coefficients; applying quantization to the extracted feature-related DWT data to obtain keys as results of the quantization; constructing a locality- sensitive fingerprint of the input data block based on the keys; computing a similarity degree between the locality-sensitive fingerprint of the input data block and a locality- sensitive fingerprint of each data block in the plurality of the data blocks in a cache memory; selecting an optimal reference data block as the data block which has the biggest similarity degree with the input data block among all data blocks in the cache memory; determining whether a differential compression is required to be applied based on the similarity degree between the input data block and the optimal reference data block; and if the differential compression is required, applying the differential compression to the input data block and the optimal reference data block. A similarity- based deduplication system is further provided.