Adaptive Data Chunking for Boundary Offset and Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data chunking algorithms, such as the fixed size chunking algorithm and the Asymmetric Extremum (AE) algorithm, are affected by boundary offset issues, leading to poor deduplication effects and increased power consumption when dealing with data segments having non-even distributions.
Innovation Solution
A variable size chunking algorithm that adapts to different data distribution characteristics by analyzing the data distribution of data segments and applying specific chunking policies based on plain or non-plain distributions, such as plateau or basin distributions, to determine optimal chunking positions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fixed size chunking algorithm is used, then the chunking process is simple and fast, but boundary offset occurs leading to poor deduplication effect
Solution Approach 1:
The patent changes the chunking parameter from fixed size to variable size based on data distribution characteristics. By analyzing whether data segments have plain or non-plain distributions, the system dynamically adjusts chunking positions to avoid boundary offset while maintaining efficient processing.
Solution Approach 2:
The patent applies different chunking strategies to different local data segments based on their distribution characteristics. Plain distribution segments use one chunking approach while non-plain distribution segments use another, optimizing deduplication for each local region without compromising overall system performance.
2Productivity
If the Asymmetric Extremum algorithm is used, then chunking throughput is improved, but power consumption increases when dealing with non-even data distributions
Solution Approach 1:
The patent dynamically changes chunking parameters based on data distribution detection. By identifying plain versus non-plain distributions, the system selects appropriate chunking policies that maintain high throughput while reducing unnecessary computational overhead and power consumption for non-even distributions.
Solution Approach 2:
The patent introduces dynamic adaptation to data distribution characteristics, making the chunking process flexible rather than static. The system continuously monitors data segments and adjusts chunking behavior in real-time, optimizing the balance between throughput and energy consumption based on actual data conditions.
3Reliability
If variable size chunking is implemented to resolve boundary offset, then deduplication effect improves, but device complexity increases
Solution Approach 1:
The patent uses parameter changes to achieve variable size chunking through a relatively simple mechanism. By detecting data distribution type and selecting from predefined chunking policies, the system avoids complex algorithms while still achieving boundary offset resolution and improved deduplication.
Solution Approach 2:
The patent segments the chunking process into distinct phases: data distribution detection, policy selection, and chunking execution. This segmentation allows each phase to be implemented independently with manageable complexity, reducing overall system complexity while maintaining effectiveness.
Data Source
Figure 1a~1c
Figure 1d~2
Figure 3a
AI summary
Embodiments of this application disclose a chunking method and an apparatus for implementing the method. According to the method provided in the embodiments of this application, a first data segment of a first length may be determined starting from a header of a to-be-chunked data flow, a data distribution characteristic of the first data segment is determined based on character values of all characters in the first data segment, and then a chunking position is determined for different data distribution characteristics by using different methods. In this way, a data flow can be better chunked, so as to enhance a deduplication effect.