Temporal Digest Cache for Deduplication Disk Access Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication methods in data storage systems repeatedly access the on-disk digest log, leading to excessive disk usage and interference with critical data storage functions, especially when handling consecutively-stored data.
Innovation Solution
Implementing an in-memory temporal digest cache that stores digests for data blocks written in a temporal pattern, allowing subsequent deduplication to occur without repeated disk access by associating other temporally-related data with the cache.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional deduplication methods repeatedly access the on-disk digest log, then deduplication functionality is maintained, but disk access overhead increases and system performance deteriorates
Solution Approach 1:
The patent pre-loads digests for consecutively-stored data blocks into an in-memory cache before they are actually needed for deduplication operations. This preliminary action stores frequently accessed digest information in memory, avoiding the need for repeated disk accesses during subsequent deduplication operations, thus resolving the contradiction between maintaining deduplication functionality and improving system performance
Solution Approach 2:
The patent introduces an in-memory digest cache as an intermediary layer between the digest log on disk and the deduplication processing logic. This intermediary cache holds recently accessed or pre-loaded digest information, allowing the system to maintain full deduplication functionality while dramatically reducing direct disk accesses, thereby improving overall system performance
2Quantity of substance
If the digest log is stored on disk, then space for large numbers of unique blocks is accommodated, but disk access interference with critical data storage functions increases
Solution Approach 1:
The patent segments the digest storage into two parts: the main digest log remains on disk to accommodate large numbers of unique blocks, while a separate in-memory digest cache holds a subset of frequently accessed digests. This segmentation allows the system to maintain large capacity for unique blocks while reducing disk access interference by serving common queries from memory
Solution Approach 2:
The patent creates copies of digest information from the on-disk digest log and stores them in an in-memory cache. These copies contain the essential digest data needed for deduplication operations, allowing the system to accommodate large numbers of unique blocks on disk while reducing harmful disk access interference by serving requests from the memory copy when possible
3Speed
If the temporal digest cache loads multiple digests into memory, then subsequent deduplication speed increases, but memory usage increases
Solution Approach 1:
The patent applies partial action by loading only a subset of digests into the in-memory cache - specifically, digests for consecutively-stored data blocks that are likely to be accessed together. This partial loading approach increases deduplication speed for the most common cases without requiring the memory resources needed to cache all possible digests, thus balancing speed improvement with acceptable memory usage
Data Source
AI summary
Embodiments are directed to techniques for implementing a deduplication system that minimizes disk accesses to an on-disk digest log when deduplicating consecutively-stored data. These techniques for performing deduplication utilize an in-memory temporal digest cache. When the on-disk digest log is accessed for a set of data and a match is found, the temporal digest cache is written with digests not only for the set of data but also for other data stored in a temporal relationship with the set of data. This temporal digest cache allows subsequent deduplication of temporally-related data to proceed faster without needing to repeatedly access the digest log on disk.