Variable Size Data Chunking via Natural Boundary Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems face inefficiencies due to the need to back up entire data sets, leading to excessive storage requirements, as changes in data result in shifts in fixed-size blocks and subsequent MD5 hash changes, making it difficult to identify and store only the necessary changed data.
Innovation Solution
The method identifies natural boundaries in data to divide it into variable-size chunks, calculates hashes for each chunk, and compares these hashes against a database to determine which chunks need to be backed up, using a combination of Adler and MD5 hashes to efficiently identify and store only changed data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If fixed size blocks are used for data backup with MD5 hashing, then data can be divided into uniform chunks for processing, but any insertion of new data causes blocks to shift making fixed size chunking ineffective for identifying changed data
Solution Approach 1:
The patent transitions from static fixed-size blocks to dynamic variable-size chunks that adapt to data changes. The system calculates rolling hashes across the entire data set and identifies natural boundaries where hash values match, allowing chunk sizes to vary dynamically based on actual data patterns rather than predetermined fixed sizes. This enables the system to maintain effective change detection even when data is inserted or modified.
2Reliability
If all data is backed up to ensure data safety, then data loss is prevented, but the amount of disk space required becomes very large
Solution Approach 1:
The patent extracts only the changed portions of data for backup by comparing rolling hashes of variable-size chunks against previously backed up data. Instead of backing up entire data sets or all fixed-size blocks, the system identifies and extracts only those chunks whose hash values have changed, storing merely these differential portions. This dramatically reduces the volume of data requiring backup storage while maintaining the ability to reconstruct complete data sets by combining unchanged chunks with newly backed up changed chunks.
3Ease of operation
If fixed size blocks are used for data backup, then data processing is simplified, but storage efficiency deteriorates due to redundant backup of unchanged data
Solution Approach 1:
The patent segments data into variable-size chunks based on natural boundaries identified through rolling hash calculations, rather than forcing uniform fixed-size segmentation. This segmentation approach allows the system to process and backup only meaningful data portions while maintaining operational simplicity through automated hash-based boundary detection. The segmentation adapts to actual data patterns, improving storage efficiency by eliminating redundant backups of unchanged data segments.
Data Source
AI summary
A system and method obtain a set of data and identify successive subsets of data within the set of data. A boundary identifying hash is calculated on a subset of data and compared with a boundary indicating value. If the calculated boundary identifying hash matches the boundary indicating value, a natural boundary is identified in the set of data.