Deduplication Service Fingerprint Routing for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems face inefficiencies in identifying duplicate data segments due to large metadata overhead and increased deduplication time when dividing files into small segments, which affects storage space utilization and processing speed.
Innovation Solution
The system employs fingerprint generation and comparison for data segments, using SHA-1 hash fingerprints to efficiently identify unique segments, reducing communication volume and time by routing similarity groups to dedicated deduplication services, and storing only unique segments in compressed format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data files are divided into small data segments for deduplication, then more duplicates can be identified and storage space is saved, but metadata overhead increases and deduplication time increases
Solution Approach 1:
The patent divides data files into data segments for deduplication processing. This segmentation enables the system to identify duplicates at the segment level rather than file level, significantly improving storage space utilization by eliminating redundant segments across multiple files.
Solution Approach 2:
The patent introduces fingerprints as an intermediary representation of data segments. Instead of comparing actual data segments directly, the system compares fingerprints (hash values) which are much smaller in size. This intermediary approach dramatically reduces metadata overhead and accelerates the deduplication process while maintaining accurate duplicate detection.
2Quantity of substance
If data files are divided into small data segments for deduplication, then more duplicates can be identified, but metadata overhead increases
Solution Approach 1:
The patent uses fingerprints as an intermediary that represents data segments. Each data segment is represented by a compact fingerprint (hash value) rather than storing or comparing the actual segment data. This reduces metadata overhead from potentially kilobytes per segment to just bytes per fingerprint, while still enabling accurate duplicate detection.
Solution Approach 2:
The patent replaces the mechanical comparison of actual data segment content with a computational hash function that generates fingerprints. This substitution transforms a complex, resource-intensive comparison operation into a simple, fast fingerprint comparison, significantly reducing metadata requirements and processing complexity.
Data Source
AI summary
Destination namespace and file copying: a namespace service receives communication of namespace update for file from file's source. and communicates namespace update for file to an access object service identified for file. The access object service receives communication of fingerprints stream, corresponding to file's segments, from file's source, and identifies sequential fingerprints in fingerprints stream as fingerprints group. The access object service identifies group identifier for fingerprints group, and communicates fingerprints group to a deduplication service associated with group identifier range including group identifier. The deduplication service identifies fingerprints in fingerprints group which are missing from fingerprint storage, and communicates identified fingerprints to the access object service, which communicates request for file's segments, corresponding to identified fingerprints, to file's source. The deduplication service receives communication of requested segments from file's source, and stores requested segments. The access object service stores namespace update for file in distributed namespace data structure.