Digest Cache for Inline Deduplication in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication methods in data storage systems can interfere with critical operations due to high disk access and lead to over-provisioning of storage space, especially when performed inline with storage requests.
Innovation Solution
Implementing a digest-caching mechanism that stores a digest of data during a read operation and uses it during a subsequent write operation to avoid creating duplicate copies, thereby improving the efficiency of inline deduplication by leveraging the proximity of read and write operations in data copy requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication is performed inline with storage requests, then storage space overprovisioning is reduced, but processing burden and disk access increase
Solution Approach 1:
The system computes and caches the digest of source data during the read phase before the write operation occurs. This preliminary computation allows the write operation to simply lookup the cached digest and compare, rather than computing a digest from scratch, thereby reducing the processing burden during the critical write path while still achieving deduplication space savings.
Solution Approach 2:
A digest cache structure is introduced as an intermediary between the read/write operations and the deduplication logic. The cache stores pre-computed digests and their corresponding logical addresses, acting as a mediator that enables fast comparison during write operations without requiring direct computation or extensive disk access, thus resolving the contradiction between space efficiency and processing speed.
2Productivity
If deduplication is performed in the background, then processing burden during write operations is reduced, but storage space overprovisioning increases
Solution Approach 1:
The system performs digest computation during the read phase (before write) and caches the result, rather than computing it during the write phase or in the background. This timing allows write operations to proceed quickly using cached digests, while still enabling space-efficient deduplication by having the digest ready for immediate comparison and potential deduplication action.
3Measurement precision
If digest computation is performed for every write operation, then deduplication accuracy is improved, but processing time increases
Solution Approach 1:
The digest is computed during the read phase and cached before the write operation occurs. During the write operation, the system performs a simple lookup of the cached digest and compares it with the new data, rather than computing a digest from scratch. This preserves duplicate detection accuracy while significantly reducing the time required during the critical write path.
Solution Approach 2:
Instead of computing the digest repeatedly during each write operation, the system creates a copy of the digest value in the cache during the read phase. This copied digest value is then reused during write operations, eliminating redundant computation while maintaining the accuracy of duplicate detection through comparison with the cached copy.
Data Source
AI summary
A technique for storing data in a data storage system detects that a read is being performed pursuant to a data copy request. In response, the data storage system stores a digest of the data being read in an entry of a digest cache. Later, when a write pursuant to the same copy request arrives, the storage system obtains the entry from the digest cache and completes the write request without creating a duplicate copy of the data.