Dynamic Deduplication Hash Table Management for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication hash tables in storage systems become excessively large due to poor deduplication rates in certain thin devices (TDevs), leading to inefficient resource allocation and increased storage costs, as they typically require entries for every 8 KB track, even when deduplication savings are minimal.
Innovation Solution
Implementing a dynamic deduplication hash table management process that purges and re-enters fingerprint entries based on specific rules, such as selecting TDevs with minimal deduplication savings and only retaining entries for larger blocks (every 512 KB to 1 MB), thereby optimizing the hash table size and reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fingerprint entries are retained for every 8 KB track to ensure complete deduplication coverage, then deduplication reliability is improved, but hash table size becomes excessively large
Solution Approach 1:
The patent applies local quality by differentiating treatment of fingerprint entries based on their deduplication potential. High-potential entries (from devices with good deduplication rates) are retained in the hash table, while low-potential entries (from devices with poor deduplication rates) are purged. This selective approach maintains deduplication reliability for important data while reducing overall hash table size.
Solution Approach 2:
The patent changes the parameter of hash table entry retention from a uniform approach (keeping all entries) to a dynamic approach based on deduplication rate metrics. By monitoring deduplication rates and adjusting retention policies accordingly, the system optimizes the balance between coverage and size.
2Measurement precision
If all fingerprint entries are maintained in the hash table to ensure accurate deduplication, then deduplication accuracy is improved, but resource allocation becomes inefficient
Solution Approach 1:
The patent implements discarding and recovering by purging fingerprint entries that are unlikely to contribute to deduplication savings (from low-potential devices) while maintaining entries from high-potential devices. This selective discarding improves resource allocation efficiency without significantly impacting deduplication accuracy for important data.
Solution Approach 2:
The patent applies partial action by not maintaining all possible fingerprint entries, but only those deemed necessary based on deduplication rate analysis. This partial retention strategy achieves sufficient deduplication accuracy while improving resource efficiency.
3Reliability
If the hash table maintains entries for all thin devices to ensure comprehensive deduplication, then deduplication completeness is improved, but storage costs increase
Solution Approach 1:
The patent applies local quality by treating different thin devices differently based on their deduplication rates. Devices with high deduplication rates receive full hash table coverage, while devices with poor rates receive reduced coverage through entry purging. This differentiated approach maintains completeness for important devices while reducing overall storage consumption.
Solution Approach 2:
The patent implements partial action by providing selective hash table coverage rather than universal coverage. This approach achieves sufficient deduplication completeness for high-potential devices while reducing total storage resource consumption through purging low-value entries.
Data Source
AI summary
A deduplication engine maintains a deduplication hash table containing a plurality of fingerprint entries identifying a track of current data and a hash value computed from the data currently stored at the identified track. A hash table manager implements a dynamic deduplication hash table management process, that includes a fingerprint entry purge process and a fingerprint entry re-entry process. The purge process is implemented on the deduplication hash table to remove selected fingerprint entries of tracks of data currently stored on the managed drives from the deduplication hash table based on a first set of Thin Device (TDev) selection rules and a second set of track selection purge rules. The re-entry process is implemented on the deduplication hash table to re-enter selected fingerprint entries into the deduplication hash table based on a third set of TDev selection rules and a fourth set of track selection re-entry rules.