Adaptive Data Chunking Using Compression-Ratio Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods face inefficiencies due to high chunking granularity, leading to increased indexing overhead and query load pressure on deduplication storage devices, and are unstable due to reliance on transmission order of multi-version data.
Innovation Solution
A content-defined multimodal data deduplication chunking method that aggregates blocks into data segments based on sample compression ratios and length ranges, using a chunking policy mapping table to select expected lengths for dividing data segments into chunks, thereby reducing the number of chunks and improving deduplication performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a finer chunking granularity is adopted to identify duplicate data, then the probability of identifying duplicate data increases and deduplication result improves, but the number of chunks increases thereby increasing indexing overhead and search complexity
Solution Approach 1:
The patent segments the data object into multiple blocks first, then aggregates consecutive blocks into data segments based on compression ratios. This two-level segmentation approach allows for fine-grained duplicate identification within blocks while maintaining coarser segments for efficient indexing and searching, thus resolving the contradiction between identification accuracy and system complexity.
Solution Approach 2:
The patent applies different processing qualities to different parts of the data by using compression ratio thresholds to determine whether to aggregate blocks into segments. High-compression-ratio regions are aggregated into segments for efficient handling, while low-compression-ratio regions are kept as individual blocks for precise duplicate detection, optimizing both identification accuracy and indexing efficiency locally.
2Measurement precision
If a finer chunking granularity is adopted to identify duplicate data, then the probability of identifying duplicate data increases, but the time efficiency of data deduplication is reduced
Solution Approach 1:
The patent performs preliminary compression ratio calculation and block aggregation into segments before the actual deduplication process. By pre-organizing data into segments based on compression characteristics, the system reduces the search space for duplicate detection, thereby improving time efficiency without sacrificing identification accuracy.
Solution Approach 2:
The two-level segmentation (blocks within segments) allows the system to process data hierarchically, first at the segment level for quick filtering and then at the block level for precise duplicate identification. This hierarchical approach significantly reduces the overall processing time compared to fine-grained chunking of the entire data object.
3Measurement precision
If content defined chunking method is used to divide data set, then duplicate data can be identified more accurately, but the number of chunks increases leading to increased indexing overhead
Solution Approach 1:
The patent merges consecutive blocks into data segments based on compression ratio characteristics. By combining multiple small blocks into larger segments where appropriate, the system reduces the total number of indexable units while maintaining the ability to identify duplicates at the block level, thus reducing indexing overhead without sacrificing identification accuracy.
Data Source
Figure 1
Figure 2
Figure 3~3e
AI summary
Embodiments of the present invention provide a data object processing method and apparatus, which can divide a data object into one or more blocks; calculate a sample compression ratio of each block, aggregate neighboring consecutive blocks with a same sample compression ratio characteristic into one data segment, and obtain the sample compression ratio of each of the data segments; and select, according to a length range to which a length of each of the data segments belongs and a compression ratio range to which the sample compression ratio of each of the data segments belongs, an expected length to divide the data segment into data chunks, where the sample compression ratio of each of the data segments uniquely belongs to one of the compression ratio ranges, and the length of each of the data segments uniquely belongs to one of the length ranges. A data object can be divided into data chunks by applying the technologies provided in the present invention.