Sequence-Based Data Chunking with Jump Scanning for Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques face inefficiencies due to boundary-shift issues in fixed-size chunking and computational overhead in content-defined chunking, leading to lower deduplication efficiency and throughput.
Innovation Solution
A sequence-based data chunking method that employs increasing and decreasing mode techniques, using jump mechanisms to skip scanning based on byte order patterns, optimizing chunk boundaries for improved throughput and deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed-size chunking is used, then the chunking process is simple and fast, but deduplication efficiency is low due to boundary-shift problems
Solution Approach 1:
The patent changes the parameter from fixed chunk size to variable chunk size determined by content characteristics. Specifically, it uses byte-order patterns (increasing/decreasing sequences) as the basis for determining chunk boundaries, transforming the chunking approach from space-based to content-based while maintaining computational efficiency.
2Reliability
If content-defined chunking (CDC) is used, then deduplication efficiency is improved, but throughput is significantly reduced due to computational overhead
Solution Approach 1:
The patent implements a jump mechanism that skips scanning of fixed amounts of bytes after detecting certain patterns (e.g., after detecting a second amount of bytes with values arranged in increasing order). This skipping approach reduces the computational overhead of CDC while maintaining its content-based chunking benefits, thereby improving throughput.
Solution Approach 2:
The patent performs partial scanning by not examining every byte in detail. Instead, it scans for specific patterns (byte order relationships) and performs jump operations, examining only necessary portions of the data stream. This partial action approach maintains deduplication efficiency while reducing overall computational burden.
3Reliability
If conventional CDC techniques with hashing or window scanning are used, then chunk boundaries are content-based, but the process is slow and results in high chunk size variance
Solution Approach 1:
The patent changes the approach from using hashing functions or window-based minimum/maximum value scanning to using byte-order pattern recognition. This parameter change transforms the computational approach from complex mathematical operations to simpler comparative operations, significantly improving processing speed while maintaining content-based chunking accuracy.
Data Source
AI summary
Disclosed herein are systems and method for data chunking. In one aspect, a method includes consecutively scanning each respective byte in a byte stream of data. The method includes in response to detecting a first amount of adjacent bytes with values arranged in a decreasing order, marking a cut-off point of a data chunk comprising at least the adjacent bytes. The method includes in response to detecting a second amount of bytes with values arranged in an increasing order, executing a jump mechanism that skips scanning of a fixed amount of bytes after the bytes with values arranged in the increasing order. The method includes subsequent to scanning the byte stream in entirety, outputting a plurality of cut-off points of identified data chunks.