Block-Based Deduplication Using Split Hash Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication methods in file systems face inefficiencies in high-speed data block processing, particularly in write operations, due to the need for frequent hash search tree searches and increased storage overhead, which also impact read and delete operations.
Innovation Solution
The implementation of a deduplication method that splits the hash value into prefix and suffix portions, storing them separately within an iNode and lookup table, allowing for reduced storage overhead and minimizing I/O operations by using the prefix as a partial address for direct lookup, thereby reducing the number of I/O operations required for deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a hash search tree is used to determine duplicate blocks, then deduplication accuracy is improved, but the number of I/O operations increases and processing speed decreases
Solution Approach 1:
The hash value is divided into two parts: a first part (prefix) and a second part (suffix). The first part is used to index into a hash table array, while the second part is used for precise matching within the selected hash table. This segmentation allows the system to quickly narrow down potential matches using the prefix, then perform accurate verification with the suffix, thereby maintaining deduplication accuracy while significantly reducing the number of I/O operations compared to searching the entire hash search tree.
2Reliability
If a hash search tree is used to store all hash values, then complete deduplication coverage is achieved, but storage overhead increases
Solution Approach 1:
The hash value is divided into a first part (prefix) and a second part (suffix). The prefix is stored in the iNode to serve as an index, while only the suffix is stored in the hash table array. This segmentation reduces the storage overhead in the hash table while maintaining complete deduplication coverage, as the prefix-suffix combination uniquely identifies each block hash.
Solution Approach 2:
The system transitions from a traditional single-dimension hash search tree structure to a two-dimensional structure where the first part of the hash value provides one level of indexing (into the hash table array) and the second part provides a second level of indexing (within the selected hash table). This dimensional change reduces storage overhead while maintaining complete deduplication coverage.
3Measurement precision
If frequent hash search tree searches are performed for write operations, then deduplication accuracy is maintained, but write operation performance deteriorates
Solution Approach 1:
The hash value is divided into a first part (prefix) and a second part (suffix). During write operations, the prefix is used to quickly index into the hash table array, and only the corresponding hash table needs to be accessed. The suffix is then used for precise matching within that specific hash table. This segmentation dramatically reduces the search space and I/O operations required during write operations while maintaining deduplication accuracy.
4Measurement precision
If the entire hash value is stored in the hash table, then lookup accuracy is improved, but storage efficiency decreases
Solution Approach 1:
The hash value is segmented into a first part (prefix) and a second part (suffix). The prefix is stored in the iNode as an index, and only the suffix is stored in the hash table array. This segmentation maintains lookup accuracy because the combination of prefix (from iNode) and suffix (from hash table) uniquely identifies the block hash, while significantly improving storage efficiency by avoiding redundant storage of the entire hash value.
Data Source
AI summary
A write-data hash value corresponding to a block of write data is generated within a deduplicating data storage system. A block of lookup table entries is retrieved from a location in a lookup table stored within a block-based storage medium, the lookup table location being indicated by a first portion of the write-data hash value and each lookup table entry including a pointer to a respective stored data volume, a portion of a hash value that corresponds to the stored data volume, and a reference count indicating a quantity of references to the stored data volume. A second portion of the write-data hash value is compared to the portions of the hash values within the block of lookup table entries, and the reference count is incremented within one of the lookup table entries for which the portion of the hash value is determined to match the second portion of the write-data hash value.