Log Structured Hash Table for Fingerprint Index Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data deduplication in storage systems faces inefficiencies due to the resource-intensive nature of maintaining large fingerprint indexes, particularly when using B-tree structures, which consume significant memory, bandwidth, and processing resources, and can lead to performance issues like bucket full conditions and update backlogs.
Innovation Solution
Implementing a log structured hash table for the fingerprint index, which reduces memory usage, bandwidth consumption, and processing resources by storing entries in buckets within blocks, allowing for efficient searching and dynamic resizing of blocks and buckets to accommodate new entries, and employing adaptive sampling to adjust the proportion of fingerprints added to the persistent index.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a B-tree structure is used for the fingerprint index, then the index can maintain sorted order and support efficient searching, but the system consumes significant memory, bandwidth, and processing resources
Solution Approach 1:
The patent creates a copy of the fingerprint index in volatile memory (cache memory) that mirrors the structure on non-volatile storage. This allows the system to serve search requests from the faster volatile memory copy, reducing the need to access and process the full index structure on storage, thereby decreasing memory bandwidth consumption and processing overhead while maintaining searching efficiency.
Solution Approach 2:
The fingerprint index is divided into multiple segments or chunks that can be independently loaded into volatile memory as needed. Instead of maintaining the entire B-tree structure in memory simultaneously, the system segments the index and loads only the relevant portions, reducing overall memory consumption while preserving efficient searching within each segment.
2Reliability
If the fingerprint index is maintained in a large B-tree structure, then comprehensive deduplication coverage is achieved, but bucket full conditions and update backlogs occur under high system loads
Solution Approach 1:
The patent implements a dynamic fingerprint index structure that can adapt its organization and capacity based on system load conditions. The index structure allows for dynamic resizing of buckets and flexible reorganization of entries, enabling the system to handle high update throughput by dynamically adjusting the index configuration rather than being constrained by a fixed B-tree structure that fills up and creates backlogs.
Solution Approach 2:
The system changes key parameters of the fingerprint index, such as bucket size, tree depth, and node capacity, based on system load and usage patterns. By dynamically adjusting these parameters, the system can optimize the balance between comprehensive deduplication coverage and update throughput, preventing bucket full conditions by increasing capacity or redistributing entries when load increases.
3Speed
If volatile memory is used for the fingerprint index, then access speed is improved, but data loss risk increases and persistent storage is required
Solution Approach 1:
The patent introduces a write-ahead log or persistent cache structure as an intermediary between volatile memory and non-volatile storage. When fingerprint index entries are modified in volatile memory for fast access, the changes are first recorded in this intermediary persistent structure. This ensures that even if volatile memory is lost, the changes can be recovered from the intermediary, providing data persistence while maintaining the performance benefits of volatile memory for active operations.
Data Source
AI summary
In some examples, a system receives, in a synchronization buffer stored in a memory, fingerprint index entries for respective data units received by the system, each fingerprint index entry of the fingerprint index entries comprising a fingerprint computed for a corresponding data unit, and a storage location indicator for the corresponding data unit. The system merges the fingerprint index entries in the synchronization buffer with fingerprint index entries of a persistent fingerprint index in a hash data structure stored in persistent storage, the hash data structure comprising a plurality of buckets.