Sketch Vector Overlap Estimation for Data Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data synchronization methods face challenges in accurately estimating the overlap between two remote data collections without incurring high communication overhead, especially when the collections are nearly identical, as they require large amounts of data transfer or compromise on precision with smaller sketch vectors.
Innovation Solution
The system divides data objects into groups, computes combined hash results for each group using multiple hash functions, and constructs a sketch vector from these results, allowing for accurate overlap estimation with reduced data transfer by comparing sketch vectors, which are generated using pair-wise independent hash functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If checksum values for all remote data objects are transferred to estimate overlap, then measurement precision is improved, but loss of information increases due to high communication overhead
Solution Approach 1:
The patent segments the remote data collection into multiple subsets, where each subset is processed independently to generate a local sketch. This segmentation allows the system to estimate overlap without transferring all checksums, thereby reducing communication overhead while maintaining measurement precision through the aggregated sketch comparison.
Solution Approach 2:
The patent creates compact sketches that are simplified copies of the original data collections. These sketches capture essential overlap information in a compressed form, allowing accurate comparison without transferring the full checksum datasets, thus reducing communication overhead while preserving measurement precision.
2Loss of information
If a small sketch vector size is used to reduce communication, then loss of information decreases, but measurement precision deteriorates for high-overlap collections
Solution Approach 1:
The patent divides the data collection into multiple subsets and generates multiple local sketches. By aggregating information from these segmented sketches, the system achieves accurate high-overlap estimation without requiring a single large sketch vector, thus reducing communication overhead while maintaining precision.
Solution Approach 2:
The patent combines multiple local sketches generated from different data subsets into a comprehensive overlap estimation. This merging approach allows the system to achieve the precision of a large sketch vector while only transmitting smaller individual sketch components, thereby reducing communication overhead.
3Measurement precision
If multiple hash functions are applied to each data object to improve overlap detection, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent applies multiple hash functions to segmented subsets of data rather than to all data objects. This segmentation strategy reduces the total number of hash computations required while maintaining detection precision, as each subset is processed independently with a manageable number of hash functions.
Solution Approach 2:
The patent applies hash functions to a representative sample of data subsets rather than exhaustively to all data objects. This partial action approach maintains sufficient detection precision for high-overlap estimation while significantly reducing the computational complexity of hash operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A versioning system determines an amount by which a first collection and a second collection of data objects overlap. The system divides the first collection of data objects into m possibly overlapping groups of average size s and computes one combined hash result for each group. The system then constructs a first sketch vector with n elements based on the combined hash results. A respective element of the first sketch vector is selected, using a selection function, from the combined hash results that are computed with the hash function corresponding to the element's index. Next, the system receives a second sketch vector for the second collection of data objects, and determines a sketch-vector overlap between the first and second sketch vectors. The system then computes a data-object overlap between the first and second collections of data objects based on the sketch-vector overlap.