Receiver-Side Data Deduplication via Fingerprint Inversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques require the sender to maintain a large fingerprint dictionary, which is inefficient and makes it difficult to apply deduplication globally across multiple data sources, and do not effectively reduce bandwidth usage between the sender and receiver.
Innovation Solution
The receiver maintains the fingerprint dictionary, and the sender sends fingerprints to the receiver to identify data units to be uploaded, allowing only necessary data to be transferred, thus reducing bandwidth usage and eliminating the need for the sender to maintain a large fingerprint dictionary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If the sender maintains a fingerprint dictionary for data deduplication, then bandwidth usage is reduced by uploading only unique data, but the device complexity and storage requirements at the sender increase significantly
Solution Approach 1:
The patent inverts the conventional approach by moving the fingerprint dictionary from the sender to the receiver. Instead of the sender maintaining and querying a local fingerprint dictionary, the receiver maintains the dictionary and performs the deduplication checks. This inversion transfers the computational and storage burden from the sender to the receiver, reducing sender complexity while maintaining deduplication benefits.
Solution Approach 2:
The patent introduces fingerprints as an intermediary mechanism. Rather than directly comparing large data blocks, the system uses compact fingerprint representations (hash values) to identify and compare data units. This intermediary allows efficient deduplication with minimal storage requirements at the receiver, enabling global deduplication across multiple senders without requiring each sender to maintain a complete fingerprint dictionary.
2Loss of energy
If the sender maintains a local cache of data for deduplication, then upload bandwidth is reduced, but the quantity of storage required at the sender increases to many gigabytes or terabytes
Solution Approach 1:
The patent inverts the storage architecture by eliminating the need for senders to maintain large local caches. Instead, the receiver maintains the fingerprint dictionary that enables deduplication decisions. This inversion shifts the storage burden from multiple senders to a single receiver, dramatically reducing the total storage requirements across the system while maintaining upload bandwidth efficiency.
Solution Approach 2:
The patent uses fingerprint copies (hash values) instead of actual data copies. Rather than storing gigabytes or terabytes of actual data locally at each sender, the system stores compact fingerprint representations at the receiver. These fingerprint copies enable deduplication decisions without requiring substantial storage capacity at senders, reducing storage requirements from GB/TB levels to minimal overhead.
3Adaptability or versatility
If multiple senders upload data to a single receiver, then data consolidation is achieved, but applying deduplication globally across all data sources becomes difficult with conventional sender-side techniques
Solution Approach 1:
The patent merges the fingerprint dictionary management function into a single location (the receiver) rather than distributing it across multiple senders. This consolidation allows the receiver to maintain a unified view of all data from multiple senders, enabling global deduplication across the entire system. The receiver can identify duplicates from any sender against the centralized dictionary, achieving versatile global deduplication without the complexity of distributed synchronization.
Solution Approach 2:
The receiver's fingerprint dictionary serves multiple functions: it enables deduplication for multiple different senders, supports global data consolidation, and provides a universal reference for all uploaded data. This single structure performs the work of multiple distributed dictionaries would need to perform, achieving adaptability and versatility in global deduplication while reducing overall system complexity.
Data Source
AI summary
Methods and apparatus for receiving uploaded data from a sender at a receiver. A data deduplication technique is described that may reduce the bandwidth used in uploading data from the sender to the receiver. In the technique, the receiver, rather than the sender, maintains a fingerprint dictionary for previously uploaded data. When a sender has additional data to be uploaded, the sender extracts fingerprints for units of the data and sends the fingerprints to the receiver. The receiver checks its fingerprint dictionary to determine the data units to be uploaded and notifies the sender of the identified units, which then sends the identified units of data to the receiver. The technique may, for example, be applied in virtualized data store systems to reduce bandwidth usage in uploading data.