In-Memory Hash Table Segmentation for Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies due to costly lookups in persistent media, leading to increased read amplification and reduced system throughput, especially when performing inline deduplication with hash tables stored on both in-memory and persistent media.
Innovation Solution
The proposed method defers deduplication lookup to the hardening process, performing inline deduplication only on in-memory cache levels and using a multi-level caching hash table to optimize deduplication by merging buckets and identifying duplicates during the hardening process, thereby reducing read amplification and improving throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deduplication lookup is performed in persistent media, then deduplication accuracy is improved, but read amplification increases and system throughput decreases
Solution Approach 1:
The hash table is segmented into multiple levels: an in-memory hash table for rapid lookups and persistent media hash tables for comprehensive deduplication. This segmentation allows the system to perform most deduplication checks in memory while using persistent media only when necessary, thereby maintaining accuracy without sacrificing throughput.
Solution Approach 2:
The in-memory hash table acts as an intermediary between the write operation and the persistent media. It intercepts deduplication lookups first, handling the majority of cases without accessing slow persistent media, thus reducing read amplification while maintaining deduplication functionality.
2Reliability
If hash table is stored on persistent media, then data durability is improved, but access time increases
Solution Approach 1:
The system segments the hash table storage into in-memory and persistent media portions. Frequently accessed hash entries are kept in memory for fast access, while less frequently accessed entries remain on persistent media for durability. This segmentation resolves the contradiction by providing both fast access paths and durable storage.
Solution Approach 2:
Different portions of the hash table have different quality characteristics: the in-memory portion provides high-speed access for hot data, while the persistent media portion provides durable storage for cold data. This local differentiation of quality allows the system to optimize for both speed and durability in appropriate contexts.
3Measurement precision
If inline deduplication is performed at all cache levels, then deduplication completeness is improved, but system performance deteriorates
Solution Approach 1:
The system performs partial inline deduplication only at the in-memory cache level, accepting that not all duplicates will be caught immediately. The remaining deduplication is completed asynchronously during the hardening process. This partial action approach maintains high system performance while achieving eventual deduplication completeness.
Solution Approach 2:
The system performs preliminary deduplication checks in the in-memory hash table before data is written to persistent media. This preliminary action catches the majority of duplicates early, and the remaining deduplication is completed as a preliminary step during the hardening process, ensuring completeness without blocking performance-critical paths.
Data Source
AI summary
A method, computer program product, and computer system for identifying, by a computing device, content in a first bucket in a first cache. It may be determined that a first portion of the content in the first bucket is a duplicate, wherein a second portion of the content in the first bucket may be unique. The first portion of the content in the first bucket may be deduplicated from the first cache. The second portion of the content may be stored in a second bucket in a second cache.