Data Segment Grouping and Routing for Faster Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face inefficiencies due to the increased metadata overhead and processing time when dividing data files into small segments, leading to fragmented storage and reduced deduplication efficiency.
Innovation Solution
A system that groups sequential data segments into initial and next sequences based on feature values, determines a similarity group identifier, and routes these segments for deduplication, using pre-generated fingerprints to minimize metadata and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data files are divided into small data segments to identify more duplicates, then deduplication ratio is improved, but metadata overhead increases and processing time increases
Solution Approach 1:
The system generates fingerprints for data segments in advance, before the actual deduplication comparison process. This preliminary fingerprint generation allows for faster comparison and identification of duplicates, reducing the overall processing time while maintaining the ability to identify duplicates in small segments
Solution Approach 2:
The patent replaces traditional byte-by-byte comparison mechanisms with fingerprint-based comparison. By substituting the mechanical comparison process with hash-based fingerprint matching, the system achieves faster deduplication processing while working with small data segments, thus reducing processing time without sacrificing deduplication effectiveness
2Loss of substance
If data files are divided into small data segments to identify more duplicates, then deduplication ratio is improved, but metadata overhead increases
Solution Approach 1:
The system merges multiple sequential data segments into groups before processing. By combining adjacent segments into larger logical units with shared metadata structures, the system reduces the total metadata overhead while maintaining the fine-grained duplicate identification capability of smaller segments
3Measurement precision
If data files are divided into small data segments, then duplicate identification accuracy is improved, but data fragmentation increases
Solution Approach 1:
The system implements a hierarchical segmentation approach where data is divided into small segments for duplicate identification, but these segments are then grouped into larger sequences. This multi-level segmentation maintains duplicate identification accuracy while reducing overall data fragmentation by creating organized structures from the smaller segments
Data Source
AI summary
System receives data stream, groups sequential data segments associated with data stream until initial sequence of data segments is formed which is larger than minimum size. System groups sequential data segments which are next after initial sequence of data segments until next sequence of data segments is formed which combined with initial sequence of data segments is larger than maximum size. System determines feature value for each data segment in next sequence of data segments. System selects value from feature values, and data segment corresponding to selected value. System divides next sequence of data segments at selected data segment into part of initial group of data segments and part of next group of data segments. System combines part of initial group of data segments with initial sequence of data segments as initial group of data segments. System routes initial group of data segments or group of corresponding fingerprints for deduplication.