Adaptive Hash Index for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data deduplication systems face performance issues due to high resource consumption when maintaining a hash index in memory, especially as they transition from backup-oriented to primary data storage, where look-ahead caches alone are insufficient to manage resource usage effectively.
Innovation Solution
The adaptive hash index service adjusts its indexing granularity and sampling strategies to balance deduplication performance and resource consumption, using a sampled compact index table and look-ahead cache to determine chunk existence and optimize indexing operations based on current resource availability and deduplication opportunities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire hash index is maintained in memory, then index serving performance is improved, but server memory resource consumption increases
Solution Approach 1:
The hash index is divided into multiple sub-indices, each stored in a separate file on secondary storage. This segmentation allows the system to load only the necessary sub-indices into memory based on current workload, rather than loading the entire index, thus reducing memory consumption while maintaining access capability.
Solution Approach 2:
The hash index is pre-built and stored on secondary storage before actual deduplication operations begin. This preliminary action allows the system to have the index ready for fast access without consuming large amounts of memory during runtime, as the index structure is already organized and stored externally.
2Device complexity
If traditional database technology such as B-tree indexing is used, then index structure is maintained, but index serving performance deteriorates due to poor locality of chunk hashes
Solution Approach 1:
Instead of using traditional B-tree indexing structures that require complex tree traversals, the system uses a direct-addressing hash table structure where each hash value maps directly to its corresponding chunk metadata. This copying approach eliminates the need for complex structural traversals and exploits the direct accessibility of hash values, significantly improving index serving performance.
3Quantity of substance
If look-ahead cache is used to reduce resource usage, then memory consumption is reduced, but it becomes inadequate for primary data storage scenarios
Solution Approach 1:
The externalized hash index system serves multiple functions: it acts as a deduplication index, a lookup table for chunk verification, and a metadata storage mechanism. This universal design allows the same index structure to be effectively used in both backup scenarios (where look-ahead caching was previously sufficient) and primary data storage scenarios (where larger working sets require external storage), making the system adaptable to different use cases.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The subject disclosure is directed towards a data deduplication technology in which a hash index service's index and/or indexing operations are adaptable to balance deduplication performance savings, throughput and resource consumption. The indexing service may employ hierarchical chunking using different levels of granularity corresponding to chunk size, a sampled compact index table that contains compact signatures for less than all of the hash index's (or subspace's) hash values, and/or selective subspace indexing based on similarity of a subspace's data to another subspace's data and/or to incoming data chunks.