Similarity-Based Data Synchronization for Low-Bandwidth Links
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data synchronization technologies face inefficiencies due to low data block repetition rates, limiting the effectiveness of deduplication methods in reducing link data over low-bandwidth connections.
Innovation Solution
Implement a similarity-based deduplication method where devices exchange similarity information and compress only the difference between data blocks, using fingerprints and addresses to identify and utilize local reference blocks for efficient data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deduplication technology is used to improve data transmission efficiency, then data links are reduced when data blocks are identical, but the technology becomes ineffective when data block repetition rates are low
Solution Approach 1:
The patent changes the parameter from exact matching to similarity-based matching. Instead of requiring data blocks to be identical for deduplication, the system uses similarity thresholds to identify partially matching blocks, allowing the deduplication technology to remain effective even when exact repetitions are rare.
Solution Approach 2:
The patent segments data blocks into smaller units and compares them individually. By dividing large data blocks into smaller segments, the system can identify partial similarities even when overall data blocks differ significantly, thereby maintaining deduplication effectiveness in low repetition rate scenarios.
2Reliability
If complete data blocks are transmitted to ensure data accuracy, then transmission reliability is improved, but data transmission volume increases over low-bandwidth links
Solution Approach 1:
The patent extracts only the differing portions of data blocks for transmission. By identifying similar data blocks at the target end and transmitting only the differences (patches), the system significantly reduces transmission volume while maintaining data accuracy through selective extraction of necessary information.
Solution Approach 2:
The patent performs preliminary similarity matching and identification of reference blocks before transmission. By pre-processing data to identify what can be reused from existing blocks, the system determines in advance what needs to be transmitted, reducing overall transmission volume while ensuring accuracy for the transmitted portions.
3Productivity
If similarity-based deduplication is implemented to reduce transmission data, then link utilization is improved, but system complexity increases due to fingerprinting and matching mechanisms
Solution Approach 1:
The patent replaces complex mechanical comparison systems with fingerprint-based identification. Instead of comparing entire data blocks byte-by-byte, the system uses cryptographic fingerprints and hash functions to quickly identify similar blocks, dramatically reducing computational complexity while maintaining similarity detection accuracy.
Solution Approach 2:
The patent introduces fingerprints and hash values as intermediary representations of data blocks. These intermediaries serve as simplified proxies that enable efficient similarity comparison without requiring direct examination of the actual data blocks, thereby reducing system complexity.
4Quantity of substance
If differential compression is applied to transmit only differences, then transmission volume is reduced, but processing time increases at both source and target ends
Solution Approach 1:
The patent replaces time-consuming differential compression algorithms with fingerprint-based identification and selective patch transmission. By using cryptographic hash functions and similarity metrics, the system quickly identifies what needs to be transmitted without performing extensive byte-level comparisons, reducing processing time significantly.
Solution Approach 2:
The patent performs preliminary fingerprinting and similarity assessment before compression. By pre-identifying similar blocks and determining the scope of differences upfront, the system avoids unnecessary compression and decompression operations, reducing overall processing time while maintaining reduced transmission volume.
Data Source
AI summary
The present disclosure relates to data transmission methods, systems, and apparatuses. In one example method, a first device sends first information of a first data block to a second device. The second device searches, based on the first information, a first index for a second data block similar to the first data block, and sends second information of the second data block to the first device. The first device locally obtains, based on the second information, a reference block that is the same as the second data block, compresses a difference part of the first data block relative to the reference block to obtain first compressed information, and sends the first compressed information to the second device. The second device obtains the first data block based on the first compressed information and the second data block.