Adaptive Data Fingerprinting for Virtual Tape Library Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication techniques face challenges in efficiently detecting and handling data block changes due to insertions or deletions, leading to inefficient storage and network resource utilization, particularly in virtual tape library systems.
Innovation Solution
A method and system that generate data fingerprints by processing data sets to detect anchors, extending the data window to ensure alignment, and using hashing algorithms like SHA-1, SHA-256, or SHA-512 to realign and store unique data segments, reducing redundancy and processing overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If fixed size data blocks are used to generate data fingerprints, then the system simplicity is improved, but the ability to detect duplicated data after insertions or deletions deteriorates
Solution Approach 1:
The patent implements dynamic block sizing where the data window size adjusts automatically based on the presence of anchors. When an anchor is detected, the window extends to include it, creating adaptive blocks that respond to data changes. This dynamic approach maintains simplicity while improving detection reliability after insertions or deletions.
Solution Approach 2:
The system changes the parameter of block size from fixed to variable based on anchor detection. By monitoring for anchors and adjusting the window size accordingly, the system transforms the static block structure into a dynamic one that can accommodate data modifications while maintaining de-duplication effectiveness.
2Reliability
If variable sized data blocks or rolling hashes are used to solve data insertion/deletion problems, then the duplicated data detection ability is improved, but the computational intensity and processing efficiency deteriorate
Solution Approach 1:
The patent segments the data processing into distinct phases: anchor detection, window extension, and fingerprint generation. This segmentation allows the system to apply variable sizing only where necessary (when anchors are present) while maintaining fixed-size processing elsewhere, balancing detection ability with processing efficiency.
Solution Approach 2:
The system applies variable window sizing partially - only when anchors are detected - rather than universally. This partial application of variable sizing maintains processing efficiency for the majority of cases while providing enhanced detection capability when needed, avoiding the full computational overhead of always using variable blocks.
3Reliability
If complete backup operations are performed in VTL systems, then data protection is improved, but the storage space utilization deteriorates due to data duplication
Solution Approach 1:
The patent uses fingerprint copying instead of full data copying. By generating and comparing fingerprints of data blocks, the system can identify duplicates and store only unique blocks, significantly reducing storage space while maintaining complete data protection through the fingerprint reference system.
Solution Approach 2:
The system changes the approach from copying entire data blocks to copying only unique blocks identified through fingerprint comparison. This parameter change in the backup operation - from complete copying to selective copying based on fingerprint uniqueness - reduces storage space utilization while maintaining data protection integrity.
Data Source
AI summary
A method and system for generating data fingerprints is used to de-duplicate a data set having a high level of redundancy. A fingerprint generator generates a data fingerprint based on a data window. Each byte of the data set is added to the fingerprint generator and used to detect an anchor within the received data. If no anchor is detected, the system continues receiving bytes until a predefined window size is reached. When the window size is reached, the system records a data fingerprint based on the data window and resets the window size. If an anchor is detected, the system extends the window size such that the window ends a specified length after the location of the anchor. If the extended window is greater than a maximum size, the system ignores the anchor. The generated fingerprints are compared to a fingerprint database. The data set is then de-duplicated by replacing matching data segments with references to corresponding stored data segments.