Dual-Window Data Chunking to Raise Throughput Without Dedup Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional content-defined chunking techniques face challenges in achieving high throughput while maintaining a high deduplication ratio, with file chunking often being the primary performance bottleneck in deduplication systems.
Innovation Solution
A jump-based content-defined data chunking method using a dual-window approach, where a larger primary window and smaller secondary window are employed to skip certain parts of the byte stream, determining chunk boundaries through a jump mechanism that shifts windows based on minimum and maximum values, enhancing throughput without compromising deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional CDC techniques are used to achieve high deduplication ratio, then deduplication efficiency is improved, but chunking throughput is significantly reduced due to additional computation overhead
Solution Approach 1:
The patent divides the scanning process into two segments: a fast jump mechanism that skips large portions of data using minimum values, and a precise marking mechanism that identifies exact cut-off points using maximum values. This segmentation allows the system to maintain high deduplication ratios while significantly improving throughput by avoiding unnecessary byte-by-byte scanning.
Solution Approach 2:
The patent implements a jump mechanism that rapidly skips through data by jumping to positions based on minimum values found in sliding windows. This skipping approach allows the system to bypass large portions of data that don't contain chunk boundaries, dramatically reducing computation overhead while still ensuring accurate chunk detection when needed.
2Productivity
If fixed-size chunking is used to simplify the process, then chunking speed is improved, but deduplication efficiency is significantly reduced due to boundary-shift problem
Solution Approach 1:
The patent transitions from static fixed-size chunking to dynamic content-defined chunking by using sliding windows that adapt to data characteristics. The window size and jump distances are dynamically adjusted based on the data being scanned, allowing the system to maintain high speed while accurately identifying chunk boundaries based on content rather than fixed positions.
Solution Approach 2:
The patent changes the parameters used for chunk boundary detection from fixed byte positions to content-based metrics (minimum and maximum values within sliding windows). This parameter change enables the system to identify meaningful chunk boundaries that reflect actual data structure, improving deduplication efficiency without sacrificing speed.
Data Source
AI summary
Disclosed herein are systems and method for executing jump-based content-defined data chunking. In one aspect, a method may generate, for a byte stream of data, a primary window and a secondary window that overlaps with the primary window. A method may scan the byte stream for data chunks using the primary window and the secondary window, wherein the scanning comprises (a) executing, based on minimum values in the secondary window when shifted along the primary window, a jump mechanism in which the primary window and the secondary window are shifted forward by a fixed amount of bytes on the byte stream and (b) marking cut-off points of detected data chunks based on maximum values within and outside the primary window. A method may output the cut-off points of the detected data chunks.