Similarity Hash for Binary Data Pages Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques fail to detect redundant data when blocks are similar but not identical, missing opportunities for deduplication due to requiring full or partial matching.
Innovation Solution
The use of a similarity hash function to identify similar data pages or blocks by computing a similarity hash value, allowing for the detection of pages or blocks with similar content even when misaligned, and enabling partial deduplication based on these similarities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data deduplication techniques require full or partial matching of data blocks, then exact duplicates can be detected and removed, but similar data blocks that are not identical cannot be detected, resulting in missed deduplication opportunities
Solution Approach 1:
The patent transforms the detection criterion from exact matching to similarity-based matching by introducing a threshold parameter. Instead of requiring data blocks to be identical, the system computes similarity scores and compares them against a threshold value, allowing flexible detection of similar blocks while maintaining controllable precision through parameter adjustment
Solution Approach 2:
The patent introduces feature vectors as an intermediary representation between raw data blocks and the matching decision. By extracting key features from data blocks and computing similarity in the feature space, the system enables efficient comparison of similar blocks without requiring exact matches, thus resolving the contradiction between detection accuracy and efficiency
2Loss of substance
If data blocks are required to be exactly matched for deduplication, then storage space can be reduced for identical copies, but similar data blocks consume additional storage space without deduplication
Solution Approach 1:
The patent modifies the matching parameter from binary (exact match/no match) to a continuous similarity score with a threshold. This allows the system to identify similar blocks that would otherwise be stored separately, reducing storage space for near-duplicates while maintaining data integrity through configurable similarity thresholds
Solution Approach 2:
The patent applies partial deduplication by identifying and removing only the redundant portions of similar data blocks rather than requiring complete matches. This approach achieves storage reduction for similar blocks without demanding full identity, thus reducing storage space while maintaining reasonable detection capability
3Adaptability or versatility
If data blocks are misaligned by integer multiples of a first value, then they may represent the same logical data, but traditional techniques cannot detect their similarity
Solution Approach 1:
The patent introduces dynamic alignment handling by allowing feature extraction and similarity computation to accommodate misaligned blocks. The system can adjust feature boundaries or use alignment-insensitive feature representations, enabling detection of similar blocks regardless of their alignment status while maintaining detection accuracy through adaptive processing
Solution Approach 2:
The patent segments data blocks into features that can be independently compared. By dividing blocks into feature units and computing similarity based on overlapping or corresponding features, the system can detect similarity between misaligned blocks by matching their respective feature segments, thus achieving both alignment tolerance and detection accuracy
Data Source
AI summary
Techniques for data processing may include: receiving a page P having a binary representation; determining a plurality of features for the page P; determining a plurality of feature hashes for the plurality of features, wherein each of the plurality of feature hashes corresponds to a different one of the plurality of features, wherein said each feature hash is determined using a hash function and in accordance with said different one of the plurality of features and an index associated with said different one of the plurality of features; and determining, in accordance with the plurality of feature hashes for the plurality of features, a similarity hash for the page P.