Block Similarity Deduplication with Bit Difference Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing block level deduplication techniques fail to maximize storage space savings while maintaining acceptable system performance, as they often miss similarities in data chunks due to fixed or inadequate block granularity, leading to increased I/O overhead and larger meta-information structures.
Innovation Solution
Implement a block similarity-based deduplication method that evaluates the bit differences between data chunks using fingerprint representations and metadata pointers, allowing for the classification of blocks as unique or similar based on predefined similarity criteria, thereby reducing redundant storage and improving deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional block level deduplication is applied with fixed block granularity, then storage space is reduced, but similar blocks with minor differences are not identified, leading to increased I/O overhead and larger meta-information structures
Solution Approach 1:
The patent changes the parameter of block comparison from exact matching to similarity-based matching with configurable thresholds. By adjusting the similarity threshold parameter, the system can identify blocks with minor differences (e.g., metadata changes, formatting variations) as duplicates, thereby improving deduplication ratio without significantly increasing I/O overhead. This resolves the contradiction by making the deduplication process more flexible and adaptive to different data patterns.
Solution Approach 2:
The patent introduces dynamic block granularity adjustment based on data patterns and access patterns. Instead of fixed block sizes, the system dynamically determines optimal block boundaries and similarity thresholds based on the specific dataset being processed. This dynamic adaptation allows the system to maximize storage savings while maintaining acceptable I/O performance by adjusting to different workloads and data types.
2Loss of substance
If block similarity criteria are relaxed to identify more similar blocks, then deduplication ratio increases, but system performance and CPU overhead worsen
Solution Approach 1:
The patent segments the block comparison process into multiple stages: first a quick fingerprint-based filter to identify potential matches, then a more detailed similarity check only for candidates that pass the initial filter. This segmentation allows the system to achieve high deduplication ratios by thoroughly checking similar blocks while minimizing CPU overhead by avoiding exhaustive comparison of all blocks. The multi-stage approach efficiently balances thoroughness with performance.
Solution Approach 2:
The patent applies partial action by performing full similarity verification only on blocks that meet preliminary similarity criteria, rather than verifying all possible block pairs. The system performs excessive action in the filtering stage (comparing fingerprints) but limits the expensive full comparison to only necessary cases. This selective verification approach achieves high deduplication ratios while controlling CPU overhead by avoiding unnecessary detailed comparisons.
3Reliability
If metadata is stored for all unique blocks, then data restoration is accurate, but meta-information structure size increases
Solution Approach 1:
The patent extracts and stores only the essential metadata needed for restoration, separating critical restoration information from optional or redundant metadata. By taking out only the necessary elements (such as block pointers, essential difference markers) and storing minimally sufficient information, the system maintains accurate data restoration capability while significantly reducing meta-information volume. This selective metadata storage resolves the contradiction between completeness and compactness.
Data Source
AI summary
Methods and apparatus are provided for block similarity based block level deduplication of data. An exemplary method comprises obtaining a deduplicated dataset comprising a plurality of unique data chunks; determining a number of differences between two of the unique data chunks; evaluating whether the number of differences satisfies a predefined similarity criteria (e.g., that the number of bit differences cannot exceed a specified limit); and storing metadata for a first one of the two unique data chunks if the predefined similarity criteria is satisfied for the two unique data chunks, wherein the metadata comprises a pointer to a second one of the two unique data chunks and bit differences between the two unique data chunks. The bit differences comprise an executable code and/or a bit mask. The predefined similarity threshold is optionally a tunable parameter. The first one of the two unique data chunks can be restored by processing the metadata.