Data Backup De-duplication via Consistent Chunking and Hash Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current de-duplication methods in data backup systems are inefficient due to the lack of effective methods to leverage the similarity between continuous versions of backup data, leading to suboptimal performance in reducing storage needs and improving data backup efficiency.
Innovation Solution
The proposed method involves predicting data chunking by searching for matched data blocks in the original backup data and using the same chunking method to chunk the current data, with hash value matching and incremental counting of continuous matched chunks to determine when to stop chunking, thereby optimizing the de-duplication process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional de-duplication methods are used without leveraging similarity between continuous backup versions, then the de-duplication process can be kept simple, but the performance of de-duplication is poor and time consumption is high
Solution Approach 1:
The patent applies preliminary action by pre-processing the original backup data into chunks and storing their hash values in a hash table before performing de-duplication on current backup data. This pre-organization of data enables faster comparison and matching operations when processing new backups, significantly reducing the time required for chunking operations while improving de-duplication performance.
Solution Approach 2:
The patent uses copying by creating hash value representations of data chunks instead of storing and comparing entire data blocks. By computing and storing hash values (cryptographic fingerprints) of original backup chunks, the system enables efficient comparison with current backup chunks without handling the actual large-volume data, thus reducing time consumption while maintaining accurate de-duplication capability.
2Quantity of substance
If the same chunking method is used for both original and current backup data, then de-duplication ratio can be improved, but the complexity of the de-duplication system increases
Solution Approach 1:
The patent applies parameter changes by systematically varying chunking parameters such as chunk size, hash algorithm selection, and threshold values for continuous matched chunks. By adjusting these parameters based on data characteristics and performance requirements, the system optimizes de-duplication ratio while managing complexity through controlled parameter variation rather than complex structural changes.
Solution Approach 2:
The patent applies segmentation by dividing backup data into discrete chunks with consistent boundaries across original and current backups. This segmentation approach, combined with independent hash computation for each chunk, enables high de-duplication ratio by identifying matching chunks while keeping the system complexity manageable through modular, independent processing of each chunk rather than complex whole-data processing.
3Measurement precision
If hash value matching is performed for every chunk, then accuracy of de-duplication is improved, but the computational overhead and processing time increase
Solution Approach 1:
The patent applies partial action by performing hash value matching selectively rather than for every single chunk. The system identifies continuous sequences of matched chunks and applies hash verification at boundary points or when discontinuities are detected, rather than verifying every chunk individually. This approach maintains high de-duplication accuracy while significantly reducing computational overhead by avoiding redundant hash computations for obviously matching continuous chunks.
Data Source
AI summary
The present invention relates to a method, system, and computer program product for data backup, the method comprising: performing first chunking on current data by using the same chunking method as that used by original backup data to obtain a current chunk; calculating hash value of the current chunk; and acquiring, from a hash value table of the original backup data, an identifier of a matched chunk whose hash value is the same as the calculated hash value of the current chunk, and incrementing number of continuous matched chunks by one. Since the pertinence between original backup data and current data is maximally utilized, performance of de-duplication method can be efficiently improved.