In-Memory Digest Cache for Deduplication Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication methods in data storage systems incur high disk access overhead due to repeated searches of the on-disk digest log, which can interfere with critical data storage functions, especially when the system is busy, and are typically performed as a background process.
Innovation Solution
Implementing an in-memory Most Wanted Digest Cache (MWDC) to store frequently deduplicated blocks, allowing for in-line deduplication of popular data blocks without accessing the on-disk digest log, by adding entries to the cache when a block is deduplicated a threshold number of times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional deduplication methods are used with on-disk digest log, then deduplication can be performed, but disk access overhead increases and performance deteriorates
Solution Approach 1:
The patent segments the digest log into two parts: a hot path in-memory cache for frequently accessed digests and a cold path on-disk digest log for less frequent access. This segmentation allows the system to serve frequent deduplication requests from memory (avoiding disk access) while maintaining the complete digest log on disk for completeness.
Solution Approach 2:
The system performs preliminary action by pre-loading frequently accessed digest entries into the in-memory cache structure before they are needed for deduplication operations. This is achieved by monitoring and caching digests that are accessed multiple times, so when similar data blocks arrive, the system can immediately find them in memory without disk access.
2Productivity
If background deduplication process is used, then system stability is maintained, but productivity is reduced due to sequential processing
Solution Approach 1:
The patent introduces dynamic behavior by allowing deduplication to occur in-line during the data ingestion process itself, rather than only as a background task. The system dynamically switches between immediate in-line deduplication (when cache hits occur) and background processing (for comprehensive deduplication), making the process adaptive to current system conditions and data patterns.
Solution Approach 2:
The patent enables continuous useful action by performing deduplication operations during the normal data ingestion flow rather than stopping for background processing. When the in-memory cache contains matching digests, the system can immediately recognize and deduplicate data blocks in-line, maintaining continuous productivity without interruption for separate background deduplication cycles.
3Loss of information
If complete digest log is stored on disk, then all data can be tracked, but device complexity increases and disk space is consumed
Solution Approach 1:
The patent extracts the most frequently accessed digest information from the main on-disk digest log and places it into a separate in-memory cache structure. This extraction removes the bulk of the complexity and disk access requirements for handling frequent deduplication cases, while the remaining on-disk log continues to track less frequent data patterns.
Solution Approach 2:
The in-memory cache structure serves as an intermediary between the data ingestion process and the on-disk digest log. It mediates by providing fast access to frequently needed digest information, reducing the direct burden on the disk-based digest log and simplifying the overall system architecture for handling common deduplication scenarios.
Data Source
AI summary
Embodiments are directed to techniques for performing deduplication. A method includes (a) obtaining a digest of a data block logically-positioned within a filesystem, the digest providing a hash value of data of the data block, (b) searching a Most Wanted Digest Cache (MWDC) within system memory for the digest, (c) locating an entry in the MWDC using the digest, wherein this locating indicates that the data block has the same data as another data block located elsewhere within the filesystem, the other data block having been previously persistently-stored, the entry having been added to the MWDC in response to the other data block having been deduplicated at least a plurality number of times, (d) locating a mapping structure referenced by the entry located from the MWDC, the mapping structure providing metadata about the other data block, and (e) deduplicating the data block and the other data block with reference to the located mapping structure.