Storage System Hash Table Management for Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication systems face a decrease in deduplication rate due to memory capacity limitations, where hashes are difficult to predict for future duplication, leading to inefficient hash table management and reduced performance.
Innovation Solution
A storage system that utilizes both memory and drive storage for hash tables, dynamically moving hashes between them based on duplication counts and update times to optimize deduplication efficiency, preventing the memory hash table from being filled with new hashes and maintaining a high deduplication rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If hashes are continuously added to the memory hash table, then the duplication determination speed is maintained, but the memory capacity is exceeded and the deduplication rate decreases
Solution Approach 1:
The hash table is segmented into two parts: a memory-resident hash table for frequently accessed hashes and a disk-resident hash table for less frequently accessed hashes. This segmentation allows the system to maintain high-speed duplication determination for common hashes while storing less critical hashes on disk to conserve memory capacity.
Solution Approach 2:
The system dynamically moves hashes between memory and disk based on their duplication counts and access patterns. When a hash's duplication count exceeds a threshold, it is moved from disk to memory to improve duplication determination speed. When memory capacity is approached, less frequently accessed hashes are moved from memory to disk, making the hash table configuration adaptive rather than static.
2Quantity of substance
If the hash table is filled with new hashes, then memory capacity is utilized, but likely-to-be-duplicated hashes are deleted resulting in decreased deduplication rate
Solution Approach 1:
The system uses duplication counts as feedback to determine which hashes to retain in memory and which to move to disk. Hashes with higher duplication counts are prioritized for retention in memory, ensuring that the most valuable duplication information remains accessible. This feedback mechanism prevents the loss of important duplication patterns while managing memory capacity.
Solution Approach 2:
The system changes the storage location parameter of hashes based on their duplication counts. By monitoring duplication counts and adjusting the storage location (memory vs. disk) accordingly, the system optimizes both memory utilization and deduplication rate, ensuring that hashes with high duplication potential are kept in memory while less critical hashes are stored on disk.
3Quantity of substance
If duplication determination is performed using only disk storage, then memory capacity is conserved, but the system load increases and performance decreases
Solution Approach 1:
Different storage media are assigned different quality characteristics: memory provides high-speed access for critical hashes, while disk provides大容量 storage for less critical hashes. This local quality differentiation allows the system to optimize performance for the most important operations while using slower, cheaper storage for less time-sensitive data.
Solution Approach 2:
The system dynamically adjusts the distribution of hashes between memory and disk based on workload characteristics and duplication patterns. During periods of high duplication activity, more hashes are kept in memory to maintain performance. When memory pressure increases or duplication rates decrease, hashes are moved to disk. This dynamic adjustment allows the system to balance memory conservation with performance maintenance.
Data Source
AI summary
When the hash of the divided data is not duplicate with the hash registered in an in-memory hash table, the hash of the divided data is registered in the in-memory non-duplication data list. When the hash registered in the in-memory non-duplication data list is duplicated with the hash registered in an on-disk hash table, a duplicate count of the duplicate hash registered in the on-disk hash table is increased by 1 and an update time of the hash is updated to the latest value. When the duplication count of the hash registered in the on-disk hash table exceeds a threshold, the hash which is registered in the on-disk hash table and in which the duplication count has exceeded the threshold is moved from the on-disk hash table to the in-memory hash table.