Selective Deduplication in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed storage systems face performance penalties due to the requirement of applying deduplication to all storage volumes, which can impact processing efficiency and lead to significant degradations in IO processing performance.
Innovation Solution
Implementing selective deduplication in distributed storage systems by distinguishing between storage volumes with high and low deduplication potential, using content-based signatures for high-dedupe volumes and non-content-based signatures for low-dedupe volumes, allowing for efficient storage of both deduplicated and non-deduplicated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication is applied to all storage volumes, then storage efficiency is improved, but IO processing performance deteriorates
Solution Approach 1:
The patent applies different deduplication strategies to different storage volumes based on their characteristics. High-dedupe volumes receive content-based deduplication with signature computation, while low-dedupe volumes receive sequential storage without deduplication. This local differentiation resolves the contradiction by optimizing each volume type appropriately rather than applying a uniform approach.
Solution Approach 2:
The system dynamically changes the deduplication parameter (content-based vs. sequential) based on the deduplication potential assessment of each storage volume. By evaluating deduplication potential and adjusting the storage approach accordingly, the system achieves both high storage efficiency for suitable volumes and high IO performance for volumes where deduplication would be detrimental.
2Quantity of substance
If content-based signatures are used for all volumes, then deduplication effectiveness is improved, but processing overhead increases
Solution Approach 1:
The patent applies content-based signature computation only partially - specifically to high-dedupe volumes where it provides value. For low-dedupe volumes, the system uses sequential storage without signature computation. This partial application of the computationally intensive deduplication process reduces overall processing overhead while maintaining deduplication effectiveness where needed.
Data Source
AI summary
For each of multiple storage volumes of a distributed storage system, it is determined whether the storage volume has a relatively high potential deduplicability or a relatively low potential deduplicability. Responsive to determining that the storage volume has the relatively high potential deduplicability, a first write flow is executed for each of a plurality of write requests directed to the storage volume, the first write flow utilizing content-based signatures of respective data pages of the storage volume to store the data pages in storage devices of the distributed storage system. Responsive to determining that the storage volume has the relatively low potential deduplicability, a second write flow is executed for each of a plurality of write requests directed to the storage volume, the second write flow utilizing non-content-based signatures of respective data pages of the storage volume to store the data pages in storage devices of the distributed storage system.