Receiver-Side Data Deduplication via Fingerprint Dictionary Inversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques require the sender to maintain a large fingerprint dictionary, which is inefficient and makes it difficult to apply deduplication globally across multiple data sources, and do not effectively reduce bandwidth usage between the sender and receiver.
Innovation Solution
The receiver maintains the fingerprint dictionary, and the sender sends fingerprints to the receiver to identify units of data to be uploaded, allowing only necessary data to be transferred, thus reducing bandwidth usage and eliminating the need for the sender to maintain a large fingerprint dictionary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the sender maintains a fingerprint dictionary for data deduplication, then duplicate data can be identified and uploaded, but the sender requires significant storage capacity and complexity to maintain the dictionary
Solution Approach 1:
The patent inverts the conventional approach by moving the fingerprint dictionary from the sender to the receiver. The receiver maintains the centralized fingerprint dictionary and performs the deduplication matching, while the sender only transmits data without needing to maintain local fingerprint storage. This resolves the contradiction by maintaining deduplication accuracy while eliminating sender storage requirements.
Solution Approach 2:
The receiver acts as an intermediary that maintains the fingerprint dictionary and performs matching operations. Instead of the sender directly comparing fingerprints locally, the sender uploads data to the receiver which then performs the deduplication check against its centralized fingerprint database. This mediator approach eliminates the need for sender-side storage while maintaining deduplication functionality.
2Reliability
If the sender uploads all data to the receiver for processing, then the receiver can identify duplicate units, but bandwidth usage is not reduced
Solution Approach 1:
The system performs preliminary action by having the receiver maintain the fingerprint dictionary in advance. When data upload is needed, the sender can efficiently determine which data units need to be uploaded by comparing against the receiver's fingerprint database (through selective transmission or pre-screening), rather than uploading all data for post-processing deduplication. This preliminary setup enables bandwidth reduction while maintaining accurate duplicate identification.
3Adaptability or versatility
If the receiver serves multiple senders, then the system can handle multiple data sources, but it becomes difficult to apply deduplication globally across all data sources
Solution Approach 1:
The receiver is designed with universal functionality to serve multiple senders while maintaining a single centralized fingerprint dictionary. The receiver performs deduplication matching for all senders against the same global fingerprint database, ensuring consistent deduplication across all data sources. This multi-functional design resolves the contradiction by enabling both multi-sender support and global deduplication consistency through the centralized architecture.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Methods and apparatus for receiving uploaded data from a sender at a receiver. A data deduplication technique is described that may reduce the bandwidth used in uploading data from the sender to the receiver. In the technique, the receiver, rather than the sender, maintains a fingerprint dictionary for previously uploaded data. When a sender has additional data to be uploaded, the sender extracts fingerprints for units of the data and sends the fingerprints to the receiver. The receiver checks its fingerprint dictionary to determine the data units to be uploaded and notifies the sender of the identified units, which then sends the identified units of data to the receiver. The technique may, for example, be applied in virtualized data store systems to reduce bandwidth usage in uploading data.