Database Duplicate Identification Using Matchcode Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face performance issues when directly comparing every record in a database to identify potential duplicates due to the need for n^2 operations, making it impractical for large datasets, and known algorithms struggle with semantic differences in records.
Innovation Solution
The method generates matchcodes for records by normalizing and reducing character strings, using wildcard characters to represent missing non-essential substrings, and comparing these matchcodes to identify matching pairs efficiently, allowing for semantic similarity assessment without the need for complex pairwise comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If every record in a database is directly compared to every other record to identify potential duplicates, then the accuracy of duplicate identification is improved, but the computational complexity and time required increase exponentially (n^2 operations)
Solution Approach 1:
The patent segments the database records into clusters using alphabetic grouping and hashing techniques. Instead of comparing every record to every other record, the system divides the dataset into smaller groups (clusters) based on initial character matching and hash values. This segmentation reduces the comparison space from n^2 to much smaller intra-cluster comparisons, resolving the contradiction between accuracy and processing speed.
Solution Approach 2:
The patent applies different comparison strategies to different parts of the data. Records are first grouped by their initial characters (local quality of the beginning of strings), then further refined using hash codes for specific positions. This localized approach allows the system to focus computational resources on promising candidate pairs while ignoring obviously dissimilar records, thereby maintaining accuracy without requiring exhaustive comparisons.
2Productivity
If optimized clustering methods are used to identify small clusters of potential duplicate records, then the processing time is reduced, but the accuracy of duplicate identification may be compromised
Solution Approach 1:
The patent performs preliminary actions by pre-computing hash codes for specific character positions and storing them in hash tables before the actual duplicate identification process. This preliminary preparation allows the system to quickly retrieve and compare only relevant records during the clustering phase, rather than performing expensive string comparisons on all record pairs. This preliminary hashing action maintains accuracy while enabling fast processing.
Solution Approach 2:
The patent introduces hash codes as an intermediary representation between the original records and the comparison process. Instead of directly comparing full record strings, the system uses hash codes of specific positions as intermediaries to quickly identify potential matches. This intermediary approach filters out non-matching records early, allowing accurate duplicate identification to proceed efficiently on a reduced dataset.
Data Source
AI summary
In accordance with the teachings described herein, systems and methods are provided for identifying potential duplicate entries in a database. Matchcodes are generated for a plurality of records, wherein a matchcode for a record may be generated by: receiving a character string from the record; determining whether the character string includes a non-essential character substring; if the non-essential character substring is missing from the character string, then generating the matchcode from the character string and adding a wildcard character to the matchcode in place of the missing non-essential character substring. The matchcodes for the plurality of records may be compared to identify matching pairs of matchcodes, wherein for the purpose of identifying a matching pair of matchcodes, two characters are considered the same if they are equal or if one or both are wildcard characters.