Hash Index Service with Look-Ahead Cache for Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Contemporary deduplication systems face throughput limitations due to the large size of chunk hash indexes, which exceed the capacity of primary storage devices like RAM, necessitating the use of slower secondary storage devices for I/O operations.
Innovation Solution
Implementing a log-based index in secondary storage with a compact index table and look-ahead cache in primary storage to reduce I/O access times, utilizing a cuckoo hash-based algorithm for collision resolution and encoding to optimize memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large hash index is maintained to identify duplicate chunks across tens of terabytes to petabytes of data, then deduplication accuracy is improved, but I/O access time increases due to the need to use slow secondary storage devices
Solution Approach 1:
The patent divides the large hash index into multiple partitions stored in secondary storage, each partition containing a subset of hash entries. This segmentation allows the system to manage the large index in manageable units and access only relevant partitions during deduplication operations, reducing I/O access time while maintaining complete deduplication coverage across tens of terabytes to petabytes of data
Solution Approach 2:
The system pre-computes and stores hash values of chunks in the segmented index structure before actual deduplication operations. By having the hash index prepared in advance with proper partitioning and metadata, the system eliminates the need for real-time hash computation during deduplication, significantly reducing I/O access time while ensuring accurate duplicate identification
2Quantity of substance
If the hash index is stored in secondary storage to accommodate large data volumes, then storage capacity is improved, but throughput decreases due to slow I/O operations
Solution Approach 1:
The patent introduces a hash table cache in primary storage (RAM) as an intermediary between the secondary storage and deduplication processing. This cache stores recently accessed or frequently used hash entries, allowing the system to maintain large storage capacity in secondary storage while achieving high throughput by serving common deduplication queries from the fast primary storage cache, eliminating the need for slow I/O operations on repeated accesses
Solution Approach 2:
The system optimizes the hash index structure by storing frequently accessed hash entries with higher priority in the primary storage cache, while less frequently accessed entries remain in secondary storage. This local quality differentiation ensures that the majority of deduplication operations can be performed with high throughput using the optimized local cache, while maintaining the ability to handle large data volumes through the full secondary storage capacity
Data Source
AI summary
The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.