Sketch Vector Overlap Estimation for High-Volume Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional relationship analysis systems are inefficient, inflexible, and inaccurate in determining overlap between high-volume digital datasets due to computationally expensive join operations and rigid approaches that fail to adapt to dataset features, leading to high error rates and poor representation of digital data.
Innovation Solution
The system employs a sketch-based sampling routine using multiple similarity estimators, such as equal bin, lesser bin, and greater bin similarity estimators, generated through permutation hashing, to dynamically estimate the intersection size of datasets by analyzing variance metrics and selecting or combining estimators for accurate overlap estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional join operations are used to determine overlap between datasets, then measurement precision is maintained, but productivity deteriorates due to computational expense
Solution Approach 1:
The patent creates compressed representations (sketches) of the original datasets that capture essential overlap information. Instead of processing entire datasets with expensive join operations, the system processes these compact sketches which are copies containing sufficient statistical information for accurate overlap estimation, thereby achieving both speed and accuracy.
Solution Approach 2:
The patent transforms the data representation by changing parameters from full dataset storage to sketch-based compressed formats with specific statistical properties. This parameter change enables faster processing while maintaining measurement precision through multiple estimators that capture different aspects of the data distribution.
2Measurement precision
If conventional single-estimator approaches are used, then device complexity is reduced, but measurement precision deteriorates due to high error rates
Solution Approach 1:
The patent merges multiple similarity estimators (e.g., MinHash, MaxHash, Cosine similarity) to produce a combined overlap estimation. Each estimator captures different aspects of data similarity, and their combination through weighted averaging or selection mechanisms reduces individual estimator errors and improves overall measurement precision.
Solution Approach 2:
The system implements a universal framework that can accommodate multiple different similarity estimators and adaptively select or combine them based on data characteristics. This multi-functional approach allows the system to handle various data types and overlap scenarios while maintaining high accuracy without requiring separate specialized systems.
3Adaptability or versatility
If conventional rigid approaches are used, then device complexity is reduced, but adaptability deteriorates as systems fail to adapt to dataset features
Solution Approach 1:
The patent implements dynamic adaptation where the system selects or weights different similarity estimators based on characteristics of the input datasets. Rather than using a fixed rigid approach, the system adapts its estimation strategy to match data features such as cardinality, distribution patterns, and overlap magnitude, thereby improving versatility.
Solution Approach 2:
The system dynamically changes parameters such as estimator selection, sketch size, and combination weights based on dataset characteristics. This parameter adaptation allows the system to optimize performance for different data scenarios without requiring completely different systems, achieving high adaptability with controlled complexity.
Data Source
AI summary
The present disclosure relates to systems, methods, and non-transitory computer-readable media that estimate the overlap between sets of data samples. In particular, in one or more embodiments, the disclosed systems utilize a sketch-based sampling routine and a flexible, accurate estimator to determine the overlap (e.g., the intersection) between sets of data samples. For example, in some implementations, the disclosed systems generate a sketch vector—such as a one permutation hashing vector—for each set of data samples. The disclosed systems further compare the sketch vectors to determine an equal bin similarity estimator, a lesser bin similarity estimator, and a greater bin similarity estimator. The disclosed systems utilize one or more of the determined similarity estimators in generating an overlap estimation for the sets of data samples.