Source-Marked Data Deduplication for Faster Cross-System Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods face challenges in meeting recovery point objectives during intensive data backup, leading to inefficient resource utilization and increased backup footprint due to inadequate policies for prioritizing online deduplication, especially when transferring data between storage systems with differing resource configurations.
Innovation Solution
A method that involves compressing and deduplicating data blocks in a source storage system, marking them with deduplication flags based on compression ratios and duplication counts, and transmitting these flagged blocks to a destination storage system for targeted deduplication, reducing unnecessary fingerprint calculations and deduplication table queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If online deduplication is performed on all incoming data blocks during intensive recovery point objective time, then data deduplication is achieved, but CPU and memory resources are overwhelmed
Solution Approach 1:
The patent segments the incoming data blocks into two categories: candidate data blocks (marked with deduplication flags) and non-candidate data blocks. Only candidate data blocks undergo fingerprint calculation and deduplication table queries, while non-candidate blocks are directly compressed and stored. This segmentation resolves the contradiction by applying deduplication resources selectively rather than uniformly to all data blocks.
Solution Approach 2:
The patent implements partial deduplication by performing deduplication operations only on a subset of data blocks that are marked as candidates based on compression ratios and duplication counts from previous operations. This partial action approach achieves adequate deduplication benefits without the excessive CPU and memory consumption that would result from processing all incoming data blocks.
2Reliability
If online deduplication is performed on all incoming data blocks, then deduplication ratio is improved, but backup speed decreases
Solution Approach 1:
The patent segments data blocks into candidate and non-candidate categories, allowing backup processing to proceed in parallel for both types. Candidate blocks undergo deduplication while non-candidate blocks are processed directly, eliminating sequential bottlenecks and improving overall backup throughput without significantly compromising deduplication effectiveness.
Solution Approach 2:
The patent performs preliminary compression on all data blocks before the backup process, and uses this pre-compression information (compression ratios and duplication counts) to pre-identify candidate data blocks that are likely to benefit from deduplication. This preliminary action allows the backup system to skip unnecessary deduplication operations during the actual backup, improving speed while maintaining deduplication ratio.
3Volume of stationary object
If deduplication is performed on all data blocks in the destination storage system, then storage space is optimized, but fingerprint calculations and deduplication table queries increase unnecessarily
Solution Approach 1:
The patent extracts and transmits only the deduplication flags along with candidate data blocks from the source storage system to the destination storage system. This extraction approach allows the destination system to identify candidate blocks without performing full fingerprint calculations on all received blocks, significantly reducing computational complexity while still achieving effective deduplication on the relevant data.
Solution Approach 2:
The source storage system performs preliminary identification of candidate data blocks based on compression ratios and duplication counts before transmission. This preliminary action embeds deduplication flags in the data blocks, enabling the destination storage system to skip unnecessary fingerprint calculations and deduplication table queries, thereby reducing device complexity while optimizing storage space.
Data Source
AI summary
Techniques for data deduplication compress and deduplicate a first set of blocks written to a source system to obtain a second set of blocks and compression ratios and numbers of duplications for blocks in the second set of blocks. Such techniques receive a request to back up the first set of blocks. Such techniques reconstruct, in response to receiving the request to back up the first set of blocks, the second set of blocks to obtain the first set of blocks. Such techniques mark, based on the compression ratios and the numbers of duplications, blocks in the first set of blocks with deduplication flags to obtain a third set of blocks. Such techniques transmit the third set of blocks to a destination system.