Compression Ratio Chunking for Data Deduplication Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods face inefficiencies due to high chunking granularity, leading to increased indexing overhead and query load pressure on deduplication storage devices, particularly when determining chunking modes and frequencies.
Innovation Solution
A data object processing method that divides data objects into blocks, aggregates blocks with similar compression ratios into segments, and selects an expected length based on compression ratio and length ranges to determine chunk boundaries, reducing the number of chunks and improving deduplication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If content defined chunking method is adopted to divide data set or data stream, then duplicate data identification accuracy is improved, but indexing overhead and searching complexity increase
Solution Approach 1:
The patent changes the parameter of chunking granularity from fine to coarse by using fixed-length blocks instead of content-defined variable-length chunks. This parameter change reduces the number of chunks and indexes, thereby lowering indexing overhead and searching complexity while maintaining acceptable duplicate identification accuracy through compression ratio-based aggregation.
Solution Approach 2:
The patent segments the data processing into two stages: first dividing data into fixed-length blocks, then aggregating these blocks into segments based on compression ratio characteristics. This segmentation approach simplifies the indexing structure compared to content-defined chunking while preserving the ability to identify duplicate content through segment-level aggregation.
2Reliability
If finer chunking granularity is used, then probability of identifying duplicate data is improved, but number of chunks increases thereby increasing indexing overhead
Solution Approach 1:
The patent merges multiple fixed-length blocks into larger segments based on compression ratio characteristics. By combining blocks with similar compression ratios, the system reduces the total number of discrete chunks while maintaining the ability to identify duplicate content, thus resolving the contradiction between duplicate identification reliability and the quantity of chunks.
Solution Approach 2:
The patent changes the granularity parameter from fine (content-defined chunks) to coarse (fixed-length blocks aggregated into segments). This parameter change directly reduces the number of chunks that need to be indexed and searched, while the compression ratio-based aggregation ensures that duplicate segments are still reliably identified.
3Loss of substance
If content defined bimodal chunking method is adopted, then space efficiency of deduplication is maintained, but query load pressure on deduplication storage device increases
Solution Approach 1:
The patent changes the chunking approach from content-defined variable-length chunks to fixed-length blocks with compression ratio-based aggregation. This parameter change simplifies the query process by reducing the number of chunks and indexes, thereby decreasing query load pressure on the storage device while maintaining space efficiency through effective duplicate segment identification.
Solution Approach 2:
The patent uses fixed-length blocks as simple, uniform units that are easier and faster to process than content-defined chunks. These blocks serve as intermediate objects that are quickly aggregated into segments, reducing the computational burden and query load on the storage system while achieving the same deduplication space efficiency.
Data Source
AI summary
Embodiments of the present invention provide a data object processing method and apparatus, which can divide a data object into one or more blocks; calculate a sample compression ratio of each block, aggregate neighboring consecutive blocks with a same sample compression ratio characteristic into one data segment, and obtain the sample compression ratio of each of the data segments; and select, according to a length range to which a length of each of the data segments belongs and a compression ratio range to which the sample compression ratio of each of the data segments belongs, an expected length to divide the data segment into data chunks, where the sample compression ratio of each of the data segments uniquely belongs to one of the compression ratio ranges, and the length of each of the data segments uniquely belongs to one of the length ranges.