Content-Based Post-Process Data Deduplication via Staging Storage Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional post-process data deduplication methods suffer from I/O amplification and increased storage space usage due to the need for temporary duplicate data copies, while inline deduplication incurs high CPU overhead and write latency, degrading I/O performance.
Innovation Solution
Implementing a content analyzer within the I/O data path to identify 'good' deduplication candidates by analyzing write data, directing them to a staging storage for post-processing, and writing 'bad' candidates directly to primary storage, thereby reducing unnecessary I/O and storage space usage without compromising deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If post-process data deduplication is used, then I/O performance is maintained, but storage space usage increases due to temporary duplicate copies
Solution Approach 1:
The patent segments the storage system into two distinct storage locations: a first storage location for storing unique data blocks and a second storage location for storing duplicate data blocks. This segmentation allows the system to maintain both deduplication benefits and performance by separating unique and duplicate data physically, avoiding the need to hold all duplicates in memory while still enabling post-process deduplication.
Solution Approach 2:
The patent introduces a content-based addressable storage layer as an intermediary between the client and the physical storage system. This intermediary layer implements the deduplication logic and manages the two storage locations, allowing post-process deduplication to occur without directly impacting client I/O performance while efficiently managing storage space through selective duplication storage.
2Quantity of substance
If inline data deduplication is used, then storage space and bandwidth utilization are optimized, but CPU overhead and write latency increase
Solution Approach 1:
The patent performs content-based addressing and deduplication determination in advance during the write operation, but defers the actual deduplication elimination action to a later post-process stage. The system preliminarily identifies duplicate blocks and stores them appropriately, then eliminates them in the background without blocking the write path, thus optimizing space utilization without incurring inline processing latency.
Solution Approach 2:
The patent extracts the computationally intensive deduplication elimination action from the inline write path and moves it to a post-process background operation. The inline path only performs lightweight content-based addressing and storage routing, while the actual deduplication (removing duplicate blocks and updating references) is performed separately afterward, reducing CPU overhead during writes.
3Productivity
If post-process data deduplication is used, then per-I/O performance penalty is avoided, but I/O amplification occurs due to additional read operations
Solution Approach 1:
The patent applies different storage qualities and locations based on data characteristics: unique data blocks are stored in the first storage location while duplicate blocks are stored in the second storage location. This local differentiation allows the system to minimize I/O amplification by storing duplicates in a dedicated location that can be managed efficiently, reducing the need to read and reprocess all data during post-processing.
Data Source
AI summary
Techniques for implementing content-based post-process data deduplication are provided. In one set of embodiments, a computer system can receive a write request comprising write data to be persisted to a storage system and can sample a portion of the write data. The computer system can further execute one or more analyses on the sampled portion in order to determine whether the write data is a good deduplication candidate that is likely to contain redundancies which can be eliminated via data deduplication. If the one or more analyses indicate that the write data is a good deduplication candidate, the computer system can cause the write data to be persisted to a staging storage component of the storage system. Otherwise, the computer system can cause the write data to be persisted to a primary storage component of the storage system that is separate from the staging storage component.