Weak Digest Deduplication for Data Block Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication systems face inefficiencies due to high resource usage and digest collisions when using strong digests, and inefficiencies in comparing data block contents, leading to increased read operations and resource consumption.
Innovation Solution
The use of a weak digest, such as Cyclic Redundancy Check (CRC), for deduplicating data blocks, which reduces the number of read operations by evaluating adjacent data blocks' digests to determine the probability of collisions, thereby improving deduplication efficiency and reducing resource requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If strong digests are used for data deduplication, then deduplication accuracy is improved, but resource usage and digest collisions increase
Solution Approach 1:
The patent segments the deduplication process into two stages: first using weak digests for initial comparison, then using strong digests only for collision resolution. This segmentation allows the system to benefit from strong digest accuracy while avoiding their high resource cost for every comparison.
Solution Approach 2:
The patent applies different quality levels of digests to different situations: weak digests are used for the general case where resource efficiency is prioritized, while strong digests are used locally only when digest collisions occur, ensuring accuracy precisely where needed.
2Measurement precision
If data block contents are compared for deduplication, then deduplication accuracy is improved, but read operations and resource consumption increase
Solution Approach 1:
The patent applies partial action by using only weak digest comparison for most data blocks, avoiding the excessive action of reading and comparing full contents. Full content comparison is performed only partially, when digest collisions are detected, thus improving overall productivity while maintaining necessary accuracy.
3Use of energy by moving object
If weak digests are used for deduplication, then resource usage is reduced, but digest collisions increase
Solution Approach 1:
The patent introduces a two-tiered intermediary system where weak digests serve as the first level of comparison (saving resources), and strong digests serve as the second level for collision resolution (ensuring reliability). This intermediary approach allows the system to achieve both resource efficiency and deduplication reliability.
4Measurement precision
If full content comparison is performed for digest collisions, then deduplication accuracy is improved, but time and resource consumption increase
Solution Approach 1:
The patent performs preliminary action by computing and storing weak digests for all data blocks before the deduplication process begins. This preliminary preparation allows the system to quickly identify potential duplicates without performing time-consuming full content comparisons, thus reducing time loss during the actual deduplication operation.
Data Source
AI summary
A method is used in deduplicating sets of data blocks. A candidate data object is identified for deduplicating a data object. A digest associated with the candidate data object matches a digest associated with the data object. Digest information of a set of data objects is evaluated. The set of data objects are selected for evaluation based on an association between location of the set of data objects and location of the candidate data object. Based on the evaluation, a deduplicating technique is applied for deduplicating the data object.