Variable Chunking Deduplication via Rolling Hash Windows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication methods, such as fixed-size partitioning, content-defined chunking, and sliding block algorithms, are inefficient for storing data with minor edits, as they store each instance of edited data, preventing effective deduplication and resulting in higher storage requirements.
Innovation Solution
A variable-sized chunking algorithm that uses a sliding window with a rolling hash function to dynamically determine cutting points, adapting to different data types and system requirements, thereby reducing storage needs and improving deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed-size partitioning is used for deduplication, then data is divided into uniform chunks for storage, but storage efficiency deteriorates when data has minor edits as each instance must be stored separately
Solution Approach 1:
The patent applies dynamics by transitioning from fixed-size chunking to variable-size chunking based on content-defined boundaries. The system dynamically determines cut points within fixed-size windows using rolling hash functions, allowing chunk sizes to adapt to actual data changes while maintaining systematic processing. This resolves the contradiction by making chunk sizes flexible rather than rigid, enabling better deduplication of edited data.
Solution Approach 2:
The patent changes the parameter of chunk size from fixed to variable based on content analysis. By using rolling hash functions to identify content boundaries within fixed windows, the system adjusts chunk sizes dynamically according to data similarity, allowing edited portions to be detected and handled appropriately rather than forcing all data into uniform chunks.
2Productivity
If content-defined chunking is used, then deduplication efficiency improves by identifying identical content, but device complexity increases due to the need for hash calculations and cut point determination
Solution Approach 1:
The patent segments the data processing task into fixed-size windows with systematic roll-over. By dividing the data stream into manageable fixed windows and applying rolling hash functions within each window, the system reduces the complexity of analyzing entire large datasets at once while still achieving content-defined chunking benefits through localized hash comparisons.
Solution Approach 2:
The patent performs preliminary hashing within fixed windows before final chunk boundary determination. By pre-calculating rolling hashes for segments of data and storing them for comparison, the system prepares data in advance for efficient deduplication matching, reducing the computational burden during the actual deduplication process.
3Measurement precision
If sliding block algorithms are used, then data with minor edits can be detected, but storage requirements increase as each edited instance must be stored separately
Solution Approach 1:
The patent uses rolling hash function copies to represent data content without storing actual data copies. By calculating and comparing hash values of data windows, the system can identify identical or similar content blocks and reference them rather than storing multiple copies, achieving space-efficient deduplication while maintaining accurate edit detection through hash comparison.
Data Source
AI summary
A computer-implemented method includes positioning a sliding window over a string of data and calculating a hash value using a rolling hash function for data encompassed by the sliding window. The method includes determining whether the calculated hash value for the data encompassed by the sliding window matches the value of an element in a cut point candidate (CPC) array by comparing the calculated hash value to the value of a first element in the CPC array. In response to the calculated hash value being equal to the value of the first element in the CPC array, the method includes defining an index of the sliding window as the cutting point for a data block.