Similarity-Based Data Synchronization for Low-Repetition Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data synchronization technologies face inefficiencies in utilizing limited link bandwidth due to low data block repetition rates, limiting the effectiveness of deduplication technologies in reducing data links.
Innovation Solution
Implementing a similarity-based deduplication method where a first device sends information to describe a data block, allowing the second device to search for similar blocks and compress only the difference parts for transmission, using weak and strong hash values to identify similar data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deduplication technology is used to improve data transmission efficiency, then data links are reduced when data blocks are identical, but data block repetition rate is low in some database scenarios so the deduplication technology cannot effectively reduce link data
Solution Approach 1:
The patent changes the parameter of data block comparison from exact matching to similarity-based matching using hash algorithms. By introducing a similarity threshold and using weak/strong hash values, the system can identify and compress similar data blocks even when they are not identical, thereby improving data transmission efficiency in scenarios with low exact repetition rates
Solution Approach 2:
The patent segments the data block comparison process into multiple stages: first using a weak hash algorithm for quick similarity screening, then using a strong hash algorithm for precise verification. This segmentation allows the system to efficiently handle large volumes of data by filtering out obviously different blocks first, then performing detailed comparison only on potentially similar blocks
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of this application disclose a data transmission method, including: A first device sends first information of a first data block to a second device; the second device searches, based on the first information, a first index for a second data block similar to the first data block; the second device sends second information of the second data block to the first device; the first device locally obtains, based on the second information, a reference block that is the same as the second data block; the first device compresses a difference part of the first data block relative to the reference block, to obtain first compressed information; the first device sends the first compressed information to the second device; and the second device obtains the first data block based on the first compressed information and the second data block. Embodiments of this application further provide a system, an apparatus, a device, and a medium. For a to-be-sent data block, if a transmitted data block is similar to the to-be-sent data block, only a difference part between the two data blocks is transmitted. This improves transmission efficiency of a data synchronization technology.