Multi-pass Duplicate Identification Using Sorted Neighborhoods
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Screening for duplicates in large databases is time-consuming and requires tedious user manipulation of search parameters, making it inefficient.
Innovation Solution
The method involves generating neighborhood records by merging data records with reference records, calculating pair distances using standard deviation scaled by a filled pairs quote value, and identifying duplicates by evaluating these distances against a threshold, allowing for user-defined parameters and keys to refine results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If window screening is used to screen for duplicates in large databases, then duplicate identification can be performed, but the process becomes time-consuming and inefficient
Solution Approach 1:
The patent divides the large database into smaller neighborhoods or blocks that can be processed independently. By segmenting the data into manageable chunks and processing them in parallel, the system maintains duplicate identification accuracy while significantly reducing the time required to screen large databases.
Solution Approach 2:
The system performs preliminary actions by pre-processing and organizing data into sorted neighborhoods before the actual duplicate screening begins. This preliminary organization allows for more efficient comparison and reduces the time needed during the actual duplicate identification process.
2Measurement precision
If traditional duplicate screening methods are used, then duplicates can be identified, but user manipulation of search parameters becomes tedious
Solution Approach 1:
The system performs self-service by automatically optimizing search parameters and conducting multiple passes of duplicate identification without requiring continuous user intervention. The automated multi-pass approach adjusts parameters dynamically, maintaining high detection accuracy while eliminating the tedious manual parameter manipulation that users would otherwise need to perform.
3Measurement precision
If comprehensive duplicate screening is performed across large databases, then accurate duplicate identification is achieved, but computer resource consumption increases
Solution Approach 1:
By segmenting the database into smaller neighborhoods that can be processed independently and in parallel, the system reduces the memory and computational resources required at any given moment while maintaining comprehensive duplicate screening coverage across the entire database.
Solution Approach 2:
The system employs multiple passes of duplicate identification, where each pass focuses on specific aspects or neighborhoods of the data. This partial action approach ensures comprehensive coverage through repeated targeted screenings rather than attempting to process everything in a single resource-intensive pass.
Data Source
AI summary
Systems and methods are provided herein for multi-pass duplicate identification using sorted neighborhoods. Data comprising a plurality of data records is received. Neighborhood records are generated by merging the plurality of data records with reference records stored in a remote data store. A resource identification field is assigned to each reference record. A pair distance, for each pair of neighborhood records having different resource identification fields, is determined by calculating a standard deviation of distances between each attribute of the pair scaled by a filled pairs quote value. Possible duplicate records are identified by evaluating each pair distance against a threshold, each possible duplicate having grouped attributes. Final duplicate records are identified by matching each group to a key.