Variable-Chunk Hash Signatures for Incremental Backup Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems face inefficiencies due to the need to back up entire data sets, even when only changes have occurred, leading to unnecessary storage of unchanged data blocks and increased disk space requirements.
Innovation Solution
The method involves dividing data into variable size chunks, calculating a first hash and an Adler hash for each chunk, and forming a signature comprising the hash, length representation, and Adler hash, which allows for efficient identification of unchanged data blocks and reduced storage needs by comparing these signatures against a database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed-size block hashing is used to identify changed data, then data backup efficiency is improved, but storage requirements increase due to inability to accurately identify changed portions
Solution Approach 1:
The patent segments data into variable-size chunks rather than fixed-size blocks. By using hash-based segmentation with rolling hashes, the system can identify natural boundaries in data changes and create smaller, more precise chunks that represent actual changed portions. This segmentation approach reduces the amount of data that needs to be stored while maintaining backup efficiency.
Solution Approach 2:
The patent changes the parameter of block size from fixed to variable. By dynamically adjusting chunk size based on actual data boundaries detected through hashing, the system optimizes both backup efficiency and storage utilization. Variable-size chunks allow the system to adapt to different data patterns and change frequencies.
2Reliability
If all data is backed up instead of only changed data, then data integrity is ensured, but data transfer volume increases unnecessarily
Solution Approach 1:
The patent extracts only the changed portions of data by comparing hash signatures of variable-size chunks between backup versions. By identifying and extracting only the minimal changed chunks rather than backing up entire files or fixed blocks, the system reduces data transfer volume while maintaining complete data integrity through the ability to reconstruct full files from chunk combinations.
Solution Approach 2:
The patent applies partial action by backing up only the necessary changed chunks rather than all data. The hashing mechanism enables identification of precise change boundaries, allowing the system to perform partial backups that are sufficient for data recovery without the excess of backing up unchanged portions.
3Quantity of substance
If variable-size chunks are used instead of fixed-size blocks, then storage efficiency is improved, but computational complexity increases due to hash calculations
Solution Approach 1:
The patent replaces complex variable-size chunk identification algorithms with simpler hash-based boundary detection. Instead of analyzing data content to find natural boundaries, the system uses rolling hashes that can be computed efficiently and compared to identify chunk boundaries. This substitution reduces computational complexity while maintaining storage efficiency.
Solution Approach 2:
The patent performs preliminary hash calculations on data during the backup process to identify chunk boundaries before actual data transfer. By pre-computing hashes and storing them in indexes, the system avoids repeated complex calculations during subsequent backup operations, reducing overall computational complexity.
Data Source
AI summary
A method comprises obtaining a plurality of variable size chunks of data, calculating a first hash for each variable size chunk of data, determining a length representation for each variable size chunk of data, determining a second hash of a subset of each variable size chunk of data, and forming a signature of each variable size chunk of data that is comprised of the first hash, the length representation and the second hash.