Composite Block File Deduplication for Modified Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication methods, particularly the fixed-size partition (FSP) method, are inefficient when files are modified, as they fail to detect redundant data blocks effectively due to changes in file length, leading to reduced deduplication rates, while variable-size chunking methods are slower and harder to implement.
Innovation Solution
A method that partitions files into composite blocks, consisting of fixed-size and variable-size blocks, where the variable-size blocks are determined based on file content, allowing for efficient detection of redundant data even after file modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the FSP method is used to partition files into fixed-size blocks, then the deduplication speed is fast, but the deduplication rate is reduced when files are modified because subsequent data blocks change and are not detected as duplicates
Solution Approach 1:
The patent segments a data block into multiple sub-blocks (first sub-block, second sub-block, third sub-block) when a modification is detected. This segmentation allows the system to identify which specific sub-blocks are modified and which remain unchanged, enabling efficient detection of redundant data while maintaining fast processing speed.
Solution Approach 2:
The patent applies different processing strategies to different parts of the data block based on local characteristics. When a modification is detected in a specific sub-block, only that sub-block and subsequent sub-blocks are reprocessed, while earlier sub-blocks retain their original deduplication status. This local quality approach maintains high deduplication rates without sacrificing overall processing speed.
2Reliability
If the CDC method or SB method is used to partition files into variable-size blocks based on content, then the deduplication rate is maintained after file modifications, but the deduplication speed becomes slower and implementation becomes more difficult
Solution Approach 1:
The patent introduces a dynamic adjustment mechanism where the block segmentation strategy changes based on detected modifications. When a modification is detected, the system dynamically segments the affected data block into sub-blocks for detailed analysis, while unchanged blocks continue to use the original fixed-size partitioning. This dynamic approach maintains high deduplication rates while preserving fast processing for unchanged data.
Solution Approach 2:
The patent changes the granularity parameter of data partitioning dynamically. Instead of using a fixed block size throughout, the system adjusts the partitioning granularity by segmenting into sub-blocks only when necessary (upon detecting modifications). This parameter change allows the system to maintain high deduplication rates for modified files while keeping the average processing speed high by avoiding unnecessary fine-grained partitioning of unchanged files.
3Ease of manufacture
If the FSP method is used, then the implementation is simple, but determining block boundaries becomes inaccurate when files are modified, causing redundant blocks to be missed
Solution Approach 1:
The patent performs a preliminary check to detect modifications in data blocks before finalizing the deduplication process. When a modification is detected, the system then performs the additional segmentation action into sub-blocks. This preliminary detection mechanism ensures that block boundaries are accurately identified for modified files without complicating the implementation for unchanged files, maintaining both simplicity and precision.
Data Source
AI summary
The present invention discloses a method for deduplication of a file, a computer program product, and an apparatus thereof. In the method, the file is partitioned into at least one composite block, wherein the composite block includes a fixed-size block and a variable-size block, the variable-size block being determined based on content of the file. Then a deduplication operation is performed on the at least one composite block.