Segment-Based De-duplication for Incremental Backups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication techniques are inefficient for incremental backups, as they often focus on full backups and do not effectively handle incremental changes, leading to high overhead and reduced throughput.
Innovation Solution
A scalable segment-based data de-duplication system that clusters fingerprints in data locality units called containers, employs varied sampling rates, and uses a per-segment summary structure to minimize unnecessary I/O operations and leverage distributed computing power for incremental backups.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional full backup de-duplication techniques are used for incremental backups, then data de-duplication ratio is maintained, but processing overhead increases and throughput decreases
Solution Approach 1:
The patent divides the backup data into segments and processes only the changed segments in incremental backups, rather than processing all data blocks. This segmentation allows the system to focus computational resources on only the portions of data that have changed, significantly reducing processing overhead while maintaining de-duplication effectiveness.
Solution Approach 2:
The patent applies partial action by performing de-duplication operations only on the necessary subset of data (incremental changes) rather than on the complete dataset. This selective processing reduces the computational burden and I/O operations required, thereby improving throughput without sacrificing de-duplication ratio.
2Quantity of substance
If sampling fingerprint index is used to fit index into RAM, then memory usage is reduced, but de-duplication accuracy decreases
Solution Approach 1:
The patent employs dynamic sampling rates that adjust based on the characteristics of the data being processed. Rather than using a fixed sampling rate, the system can increase sampling intensity when dealing with potentially duplicate data and reduce it when data is likely unique, thereby maintaining accuracy while optimizing memory usage.
Solution Approach 2:
The patent changes the sampling parameter (sampling rate) adaptively based on data characteristics and system state. This allows the system to optimize the balance between memory consumption and de-duplication accuracy by adjusting the sampling intensity rather than using a static sampling approach.
3Productivity
If per-segment summary structure is implemented, then unnecessary I/O operations are reduced, but data structure complexity increases
Solution Approach 1:
The patent performs preliminary processing by creating summary structures for segments before the actual de-duplication comparison. These summaries contain pre-computed information that allows the system to quickly determine whether segments are identical without performing full comparisons, thereby reducing I/O operations and improving efficiency.
Solution Approach 2:
The per-segment summary structure acts as an intermediary between the raw segment data and the de-duplication comparison process. This intermediate representation allows the system to make quick decisions about segment equivalence without directly comparing all underlying data, reducing the need for unnecessary I/O operations.
4Productivity
If distributed de-duplication is implemented across multiple nodes, then processing capacity increases, but coordination overhead increases
Solution Approach 1:
The patent divides the de-duplication workload into segments that can be independently processed by different nodes in the distributed system. Each node handles specific segments, and the segmentation allows for parallel processing while minimizing the coordination required between nodes, as each node operates semi-independently on its assigned segments.
Data Source
AI summary
A system in accordance with exemplary embodiments may provide a scalable segment-based data de-duplication for incremental backups. In the system, a master device on a secondary-storage node side may receive at least incremental changes, fingerprints, mapping entities, and distribute de-duplication functionality to at least a slave device, and performs data de-duplication on said plurality of segments via a way to cluster a plurality of fingerprints in a data locality unit called container for the incremental changes, varied sampling rates of a plurality of segments by having a fixed sampling rate for stable segments and by assigning a lower sampling rate for a plurality of unstable target files of de-duplication, and a per-segment summary structure to avoid unnecessary I/Os involved in de-duplication.