De-duplication Data Bank Using Pre-generated Hash Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for data transfer over networks are inefficient when dealing with large data sets, especially in situations where no data has been transferred initially, or when data cannot be compressed, as they do not address unique data transfer needs effectively.
Innovation Solution
The method involves generating and maintaining a database of fingerprints for data chunks at the destination site, allowing for re-use of data chunks and reducing network usage by transferring only unique data, using hash signatures and varying traversal patterns to represent a larger dataset than physical storage, and generating fingerprints pre-emptively or using random data if necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression algorithms are used to reduce data size, then network bandwidth utilization is improved, but data transfer speed and efficiency deteriorate when data cannot be compressed
Solution Approach 1:
The patent pre-generates hash signatures for all possible data chunks and stores them in a database before actual data transfer occurs. This preliminary action enables the system to quickly identify and reuse existing data without performing compression or hash computation during the transfer process, thus maintaining high transfer efficiency regardless of data compressibility
Solution Approach 2:
Instead of transferring actual data chunks over the network, the patent transfers only hash signatures (fingerprints) which are much smaller in size. The actual data is either reused from existing storage or generated locally at the destination, eliminating the need for network compression while maintaining transfer efficiency
2Loss of energy
If source-side de-duplication is used to remove duplicate data, then network bandwidth usage is reduced, but data transfer efficiency deteriorates when no data has been transferred previously
Solution Approach 1:
The system pre-generates and stores hash signatures for all possible data chunks in a database before any data transfer occurs. This preliminary preparation enables the destination to immediately determine whether data needs to be transferred by comparing incoming hash signatures against the pre-generated database, eliminating the need for source-side de-duplication and maintaining high transfer efficiency even for initial data transfers
Solution Approach 2:
The patent introduces a hash signature database as an intermediary between source and destination. Instead of directly comparing data chunks or performing source-side de-duplication, the system uses pre-computed hash signatures as a mediator to determine data transfer requirements, significantly reducing network bandwidth usage while maintaining transfer efficiency
3Productivity
If a database of fingerprints is maintained at the destination, then data transfer efficiency is improved, but device complexity increases
Solution Approach 1:
The patent stores only hash signatures (fingerprints) in the destination database rather than actual data chunks. These hash signatures are computationally derived representations that occupy minimal space and can be quickly compared, reducing the complexity of the database structure while maintaining high data transfer efficiency through rapid fingerprint matching
4Loss of energy
If hash signatures are used to represent data chunks, then network bandwidth usage is reduced, but measurement precision deteriorates due to potential hash collisions
Solution Approach 1:
The system pre-generates hash signatures for all possible data chunks before data transfer. By having the complete set of expected hash signatures available in the destination database, the system can definitively determine whether data already exists, eliminating false negatives that could occur with hash collisions. This preliminary preparation ensures accurate data identification while maintaining the bandwidth efficiency of hash-based representation
Data Source
AI summary
Facility for transferring data over a network between two network endpoints by transferring hash signatures over the network instead the actual data. The hash signatures are pre-generated from local static data and stored in a hash database before any data is transferred between source and destination. The hash signatures are created on both sides of a network at the point where data is local, and the hash database consists of hash signatures of blocks of data that are stored locally. The hash signatures are created using different traversal patterns across local data so that the hash database can represent a larger dataset then the actual physical storage of the local data. If no local data is present, then arbitrary data is generated and then remains static.