Data Deduplication Fingerprint Comparison Method
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication methods incur high overhead due to the need for bit-by-bit comparison of data blocks when hash algorithm collisions occur, leading to reduced system performance.
Innovation Solution
The method generates fingerprints for data blocks using two different hash algorithms, comparing these fingerprints to determine duplicates, thereby reducing the overhead of identifying duplicate data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If bit-by-bit comparison is performed to verify duplicate data blocks when hash collision occurs, then accuracy of duplicate detection is improved, but system performance deteriorates due to high overhead
Solution Approach 1:
The patent segments the duplicate detection process into two stages: first uses a fast hash algorithm for initial filtering, then only performs bit-by-bit comparison on cases where hash collision occurs. This segmentation reduces the overall computational overhead while maintaining detection accuracy.
Solution Approach 2:
The patent introduces an intermediary mechanism (fingerprint database with hash indices) between the data blocks and the comparison process. This intermediary structure enables efficient retrieval and reduces the need for exhaustive bit-by-bit comparisons, thereby improving system performance while maintaining accuracy.
2Reliability
If decompression is performed to avoid misjudgment caused by hash collision, then reliability of duplicate detection is improved, but time consumption increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing fingerprints (hash values) of data blocks in a fingerprint database before actual duplicate detection. This preliminary indexing allows for rapid comparison and reduces the need for time-consuming decompression operations during detection.
Solution Approach 2:
The patent changes the parameter representation by using hash fingerprints as intermediaries instead of directly comparing raw data blocks. This parameter transformation enables faster comparison operations while maintaining detection reliability through the use of cryptographic hash functions.
Data Source
AI summary
Techniques for managing data blocks involve: generating, based on a first hash algorithm, a first fingerprint for a first block. The techniques further involve: if it is determined that there is a second fingerprint, in a fingerprint database, that is generated for a second block based on the first hash algorithm and matches the first fingerprint, determining whether there is a third fingerprint, in the fingerprint database, that is generated for the second block based on a second hash algorithm. The techniques further involve: if it is determined that the third fingerprint exists in the fingerprint database, generating a fourth fingerprint for the first block based on the second hash algorithm; and determining whether the first block and the second block are duplicate by comparing the third fingerprint and the fourth fingerprint. Such techniques can effectively reduce the overhead of identifying duplicate data blocks in data deduplication.