Sampled Anchor Matching for Scalable Duplicate Data Elimination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication techniques, such as those using rolling hash functions and fixed-size blocks, face performance issues due to computationally intensive hashing calculations and scalability limitations, especially with large data sets like terabytes or petabytes, and are not scalable for large data environments.
Innovation Solution
A sampling-based technique that identifies anchors within a data set through rolling hash or contextual information, stores these anchors in a database, and performs bit-by-bit comparisons to replace duplicate data with storage indicators, reducing storage space and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rolling hash function is used to generate sub-blocks and determine duplicates, then data de-duplication can be achieved, but computationally intensive hashing calculations affect system performance
Solution Approach 1:
The patent segments the data de-duplication process into two phases: a sampling phase that uses rolling hash to identify potential duplicate anchors, and a verification phase that performs bit-by-bit comparison only on identified candidates. This segmentation reduces the number of computationally intensive operations while maintaining de-duplication accuracy.
Solution Approach 2:
The patent applies partial action by using rolling hash not on the entire data set but only on sampled anchors (e.g., every Nth block or at specific positions). This partial sampling approach significantly reduces computational overhead while still enabling effective duplicate detection through subsequent verification of identified anchors.
2Reliability
If hash table is used to store and compare hash values, then duplicate detection is enabled, but hash table size increases with data set size and does not scale to large data sets
Solution Approach 1:
The patent extracts only the essential identifying information from data blocks (anchors) rather than storing complete hash tables for all data. By sampling and storing only anchor points with their hash values, the system reduces memory requirements and improves scalability while maintaining duplicate detection capability through verification against the sampled anchors.
3Ease of manufacture
If fixed size blocks are used for de-duplication, then processing is simplified, but single bit addition causes misalignment and prevents de-duplication
Solution Approach 1:
The patent introduces dynamic anchor selection that can adapt to data changes. Instead of rigid fixed-size blocks, the system identifies anchors at variable positions based on sampling criteria (e.g., every Nth block, or at boundaries defined by rolling hash). This dynamic approach maintains processing simplicity while preventing misalignment issues caused by single bit additions.
4Reliability
If complete backup operation is performed to VTL system, then data protection is ensured, but storage space is inefficiently consumed due to duplication
Solution Approach 1:
The patent performs preliminary de-duplication processing before data is written to the VTL system. By identifying and eliminating duplicate anchors and their associated data during the backup operation itself, the system ensures data protection while preventing inefficient storage space consumption from the outset.
Data Source
AI summary
A technique for eliminating duplicate data is provided. Upon receipt of a new data set, one or more anchor points are identified within the data set. A bit-by-bit data comparison is then performed of the region surrounding the anchor point in the received data set with the region surrounding an anchor point stored within a pattern database to identify forward/backward delta values. The duplicate data identified by the anchor point, forward and backward delta values is then replaced in the received data set with a storage indicator.