Revision-Tolerant Data De-duplication Using Fixed-Length Handles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data de-duplication methods are computationally intensive and impractical for high-bandwidth streaming data due to the need for near-perfect hashing, leading to dictionary misses and inefficiencies in handling minor data modifications.
Innovation Solution
A revision-tolerant data de-duplication architecture that uses fixed-length 'handles' for comparison instead of near-perfect hashing, allowing for efficient matching and alignment of data segments, enabling tree-based search techniques and reducing computational demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If near-perfect hashing functions (e.g., SHA-1, MD5) are used to avoid hash collisions, then hashing accuracy is improved, but computational complexity increases significantly
Solution Approach 1:
The patent divides the data segment into multiple fixed-size sub-segments and extracts a handle (fingerprint) from each sub-segment. Instead of computing a single hash for the entire segment, the system processes smaller units independently, reducing the computational burden per operation while maintaining overall accuracy through multiple comparison points.
Solution Approach 2:
The patent extracts a fixed-size handle (fingerprint) from each sub-segment of the data. This handle serves as a simplified representation that can be quickly compared in the hash table, avoiding the need to compute and compare full near-perfect hashes of entire segments, thus reducing computational complexity while preserving matching accuracy.
2Measurement precision
If strong hashing functions are used to ensure unique segment identification, then de-duplication accuracy is improved, but processing speed decreases
Solution Approach 1:
By segmenting the data into fixed-size sub-segments and computing handles for each, the system enables parallel processing and faster lookup operations. The smaller handle size allows for quicker hash table comparisons while multiple handles per segment maintain de-duplication accuracy even when data modifications occur.
Solution Approach 2:
The patent computes handles for multiple sub-segments within a single data segment, which may seem excessive at first glance. However, this partial redundancy ensures that if one handle fails to match due to data modification, other handles provide fallback matching opportunities, maintaining accuracy while the fixed-size nature of handles keeps processing fast.
3Manufacturing precision
If breakpoint identification is performed for each byte to define content-defined segments, then segment accuracy is improved, but computational demand increases
Solution Approach 1:
The patent replaces complex byte-level breakpoint identification with a simpler fixed-size segmentation approach. Each segment is divided into equal-sized sub-segments, and handles are extracted from these uniform units. This eliminates the need for computationally intensive polynomial division and breakpoint detection algorithms while maintaining sufficient segmentation quality for de-duplication purposes.
4Measurement precision
If content-defined segments are used instead of fixed-size segments, then data matching precision is improved, but processing complexity increases
Solution Approach 1:
The patent uses fixed-size segmentation with multiple handles per segment as a practical compromise. While not perfectly content-defined, the fixed-size approach with multiple comparison points (handles) achieves sufficient matching precision for most de-duplication scenarios while dramatically reducing processing complexity compared to true content-defined segmentation.
Data Source
AI summary
Redundant data is removed from a volume of data by partitioning the volume of data into fixed-length input segments and, for each of the input segments, traversing nodes of a search tree in accordance with the value of a fixed-size portion of the input segment to determine if the search tree contains a pointer to a matching fixed-sized portion of a segment within a dictionary. If the search tree contains the pointer, the input segment is compared with the segment within the dictionary pointed to by the pointer, and a token representative of the segment within the dictionary is substituted for at least part of the input segment determined to match the segment within the dictionary.