Receiver-Maintained Fingerprint Dictionary for Data Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data deduplication techniques require the sender to maintain a large fingerprint dictionary, which is inefficient and makes it difficult to apply deduplication globally across multiple data sources, and do not effectively reduce bandwidth usage between the sender and receiver.

Innovation Solution

The receiver maintains the fingerprint dictionary, and the sender sends fingerprints to the receiver to determine which data units to upload, allowing only necessary data to be transmitted, thus reducing bandwidth usage and eliminating the need for the sender to maintain a large fingerprint dictionary.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If the sender maintains a large fingerprint dictionary to identify data units for upload, then data deduplication can be performed at the sender, but the device complexity and storage requirements at the sender increase significantly

Engineering Contradiction:

Improvedata deduplication capabilityVSAvoidfingerprint dictionary storage

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent inverts the conventional approach by moving the fingerprint dictionary from the sender to the receiver. Instead of the sender maintaining and querying a large fingerprint dictionary to identify which data units to upload, the receiver maintains the fingerprint dictionary and sends back information about which data units are needed. This inversion eliminates the need for the sender to store large amounts of fingerprint data while preserving the deduplication functionality.

Inventive Principle:

Principle #13The other way round (Inversion)

2Reliability

If the sender uploads all data to the receiver for processing, then the receiver can identify units of data to be processed, but bandwidth usage between sender and receiver is not reduced

Engineering Contradiction:

Improvedata processing accuracyVSAvoidbandwidth usage

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent applies preliminary action by having the receiver prepare and send back information about which data units are needed before the sender uploads any data. The receiver queries its fingerprint dictionary with the sender's data units, determines which units are actually needed, and communicates this back to the sender. This allows the sender to upload only the necessary data units, avoiding the waste of bandwidth that would occur if all data were uploaded first and then filtered.

Inventive Principle:

Principle #10Preliminary action

3Loss of energy

If the sender identifies and uploads only necessary data units, then bandwidth usage is reduced, but the sender must maintain a large fingerprint dictionary

Engineering Contradiction:

Improvebandwidth usageVSAvoidfingerprint dictionary storage

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The patent resolves this contradiction by inverting who maintains the fingerprint dictionary. Instead of the sender maintaining it, the receiver maintains the fingerprint dictionary and uses it to determine which data units are needed. The receiver sends this information back to the sender, allowing the sender to upload only necessary data without having to store large amounts of fingerprint data locally.

Inventive Principle:

Principle #13The other way round (Inversion)

4Loss of energy

If deduplication is applied locally at each sender, then each sender can reduce its own bandwidth usage, but global deduplication across multiple data sources cannot be achieved

Engineering Contradiction:

Improvebandwidth usageVSAvoidglobal deduplication capability

Core Design Contradiction:

Loss of energyVSAdaptability or versatility

Solution Approach 1:

The patent applies universality by making the receiver serve multiple senders while maintaining a single centralized fingerprint dictionary. The receiver can process deduplication queries from multiple different senders using the same fingerprint database, enabling global deduplication across all data sources. This multi-functional approach allows the same receiver infrastructure to serve multiple clients while maintaining consistent deduplication policies across all of them.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS9794191B2Reduced bandwidth data uploading in data systems

Publication Date: 2017.10.17 AMAZON TECH INC

AI summary

Methods and apparatus for uploading data from a sender to a receiver. A data deduplication technique is described that may reduce the bandwidth used in uploading data from the sender to the receiver. In the technique, the receiver, rather than the sender, maintains a fingerprint dictionary for previously uploaded data. When a sender has additional data to be uploaded, the sender extracts fingerprints for units of the data and sends the fingerprints to the receiver. The receiver checks its fingerprint dictionary to determine the data units to be uploaded and notifies the sender of the identified units, which then sends the identified units of data to the receiver. The technique may, for example, be applied in virtualized data store systems to reduce bandwidth usage in uploading data.