Multi-Bin Record Pairing for Scalable Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication solutions face challenges in efficiently identifying all similar record pairs in large datasets due to performance limitations and high processing times, particularly in handling records with minor variations and scalability issues.
Innovation Solution
A system that uses binning to assign records to bins based on local and global similarity functions, generating Global Candidate Pairs with a controlled false negative budget, and employing classifiers to filter duplicates, allowing for efficient computation and scalability across millions of records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive enumeration of all record pairs is performed to identify similar records, then all similar record pairs can be detected, but processing time becomes prohibitively long and performance deteriorates
Solution Approach 1:
The patent divides the data into multiple bins based on local similarity functions, so that similar records are grouped together in the same bins. This segmentation allows the system to only compare records within the same bin rather than performing exhaustive enumeration of all record pairs, thereby reducing processing time while maintaining detection completeness.
Solution Approach 2:
The patent performs preliminary binning of records based on local similarity functions before conducting the actual similarity comparison. This preliminary action organizes records into bins in advance, so that when global similarity evaluation is needed, only records within the same bin need to be compared, significantly reducing the search space and processing time.
2Productivity
If traditional binning methods are used to reduce processing time, then performance improves, but false negatives increase and detection accuracy deteriorates
Solution Approach 1:
The patent introduces a two-level binning structure with local bins and global bins, adding an additional dimension to the traditional single-level binning. Local bins are created based on local similarity functions for fast initial grouping, while global bins are created based on global similarity functions to ensure that similar records are not separated. This multi-dimensional approach maintains detection accuracy while enabling parallel processing for improved productivity.
Solution Approach 2:
The patent uses local similarity functions as intermediaries to create local bins that guide the initial grouping of records. These local bins serve as a filtering mechanism that reduces the search space for global similarity evaluation, while the global similarity function acts as a mediator to verify and refine the grouping, ensuring that false negatives are minimized.
3Device complexity
If single-token indexing is used for similarity search, then implementation is simple, but scalability to large datasets deteriorates
Solution Approach 1:
The patent segments the indexing process into multiple levels: local indexing based on local similarity functions and global indexing based on global similarity functions. Each level uses multiple tokens rather than single tokens, creating a hierarchical index structure that is more complex to implement but scales much better to large datasets by enabling parallel processing and reducing the search space at each level.
Data Source
AI summary
A very efficient computer system is presented to generate all pairs of records that have a certain similarity. Similarity is defined in terms of the textual similarity of the record attributes and/or absolute difference for numeric record attributes. Software assigns each record to a number of bins, and then compares pairs of records that belong to the same bin. This is more efficient than comparing all pairs of records since the number of records compared to each other is much smaller.