Two-Stage Content-Defined Chunking for Parallel Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Variable-size deduplication in data storage networks faces performance issues due to the compute-intensive nature of sliding-window based hash processes in content-defined chunking, leading to poor deduplication ratios and efficiency when performed sequentially or in parallel without proper optimization.
Innovation Solution
A two-stage content-defined chunking method is implemented, where the input data stream is partitioned into equal-size segments, and hash computations are performed in parallel using multiple threads, with chunk boundaries determined using advanced vector extensions (AVX) to efficiently identify unique chunks and improve deduplication ratios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sliding-window based hash process is performed sequentially for content-defined chunking, then chunk boundary accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The input data stream is divided into multiple equal-size segments that can be processed independently in parallel. Each segment is handled by a separate processing unit, allowing concurrent execution of hash computations while maintaining accurate chunk boundary detection within each segment context.
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing by adding the dimension of concurrent execution. Multiple processing units operate simultaneously on different segments, dramatically improving throughput while preserving the accuracy of chunk boundary identification through proper segment boundary management.
2Productivity
If parallel content-defined chunking is used to improve processing speed, then productivity is improved, but deduplication ratio deteriorates due to forced chunk cuts at segment boundaries
Solution Approach 1:
The data stream is pre-partitioned into equal-size segments before parallel processing begins. This preliminary segmentation allows each processing unit to work on fixed-size chunks independently, enabling parallel execution while maintaining consistent processing boundaries that prevent deduplication ratio degradation.
Solution Approach 2:
Each processing unit applies the content-defined chunking algorithm with local quality optimization, processing its assigned segment independently with appropriate hash computation and comparison parameters. This localized processing ensures that chunk boundaries are accurately determined within each segment context while maintaining overall deduplication effectiveness.
3Productivity
If fixed-size chunking is used to improve processing speed, then productivity is improved, but deduplication ratio deteriorates due to boundary shifts from insertions and deletions
Solution Approach 1:
The patent implements dynamic chunk sizing within parallel processing by allowing chunk boundaries to be determined by content characteristics rather than fixed positions. Each processing unit dynamically identifies chunk boundaries based on hash value comparisons, enabling adaptive chunk sizes that maintain high deduplication ratios while preserving parallel processing performance benefits.
Data Source
AI summary
Methods, non-transitory machine readable media, and computing devices that compare a hash value to a predefined value for sliding windows in parallel for segments partitioned from an input data stream. A bit array is parsed according to minimum and maximum chunk sizes to identify chunk boundaries for the input data stream. The bit array is populated based on a result of the comparison and portions of the bit array are parsed in parallel. Unique chunks of the input data stream defined by the chunk boundaries are stored in a storage device. Accordingly, this technology utilizes parallel processing in two stages. In a first stage, rolling window based hashing is performed concurrently to identify potential chunk boundaries. In a second stage, actual chunk boundaries are selected based on minimum and maximum chunk size constraints. This technology advantageously facilitates significant deduplication ratio improvement as well as improved parallel chunking performance.