Truncated Fingerprint Deduplication for Hash Table Size Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems face inefficiencies due to large hash table sizes, which increase computational costs and storage requirements, and fail to recognize misaligned duplicate data chunks written sequentially in multiple physical locations.
Innovation Solution
The method involves calculating hash values for unwritten data blocks, iteratively truncating them using two functions, and comparing the truncated values to a hash table to identify sequential matches, allowing for deduplication of misaligned data chunks across different storage locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full hash values (20-32 bytes) are stored in hash tables to ensure unique identification of data chunks, then deduplication accuracy is improved, but storage overhead and computational costs increase
Solution Approach 1:
The patent segments the hash value comparison process into multiple stages: first comparing a prefix portion of the hash value, then progressively comparing additional portions only when needed. This segmentation allows the system to quickly eliminate non-matching entries with minimal comparison overhead while ensuring complete accuracy for potential matches.
Solution Approach 2:
The patent applies partial action by performing incomplete hash value comparisons initially - only comparing portions of the hash values rather than the full 20-32 byte values. This partial comparison is sufficient for most cases to determine non-matches, and only full comparisons are performed when necessary, reducing overall computational effort.
2Reliability
If larger hash values are used to uniquely represent data chunks, then false positives are reduced, but search time and processing speed decrease
Solution Approach 1:
The patent divides the hash value into multiple segments or portions, comparing them in a staged manner. The comparison starts with the first portion of the hash value, and only proceeds to subsequent portions if the previous portions indicate a potential match. This segmentation maintains reliability by ensuring complete verification while improving speed through early termination of non-matching comparisons.
Solution Approach 2:
The system performs partial hash value comparisons rather than complete comparisons for all entries. By comparing only the necessary portions of hash values (often just the first portion), the system achieves sufficient reliability to eliminate false positives in most cases while dramatically reducing search time.
3Productivity
If fine-grained deduplication is performed at smaller block sizes (e.g., 4 KB), then deduplication effectiveness is improved, but metadata requirements increase significantly
Solution Approach 1:
The patent segments the hash value comparison into progressive stages, where each stage compares a portion of the hash value. This segmentation allows fine-grained deduplication to be performed efficiently by quickly eliminating non-matching small blocks without requiring extensive metadata processing, as the staged comparison reduces the computational burden per block.
Solution Approach 2:
The system performs partial hash comparisons for fine-grained deduplication, comparing only the necessary portions of hash values for small blocks. This partial action approach maintains high deduplication effectiveness at fine granularities while reducing metadata overhead by avoiding complete hash value comparisons for all block pairs.
4Measurement precision
If complete hash value comparisons are performed for all data blocks, then deduplication accuracy is maintained, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the hash value comparison process into multiple progressive stages, comparing different portions of hash values in sequence. This segmentation reduces computational complexity by breaking down the complex task of complete hash comparison into simpler, staged comparisons that can be terminated early when mismatches are detected.
Solution Approach 2:
The system performs partial hash value comparisons rather than complete comparisons for all data blocks. By comparing only the necessary portions of hash values (often just the first portion, and sometimes only a fraction thereof), the system maintains sufficient deduplication accuracy while dramatically reducing computational complexity and processing time.
Data Source
AI summary
The system, devices, and methods disclosed herein relate to data ratio reduction technology adapted to reduce storage costs by weeding out duplicative data write operations. The techniques and systems disclosed achieve deduplication benefits by reducing the size of hash values stored hash tables used to compare unwritten data blocks to data that has already been written and stored somewhere in physical storage. The data deduplication systems, methods, and products facilitate deduplication at the block level as well as for misaligned data chunks within data blocks, that is an unwritten data block that has been stored sequentially in two different physical locations. The deduplication teachings herein are amenable to varying data block sizes as well as data chunk sizes within blocks. Our embodiments enhance computer performance by substantially reducing computational speeds and storage requirements attendant to deduplication systems using larger hash table data sizes.