Bulk Deduplication Detection via Record Clustering and Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face significant performance issues when identifying and removing duplicate records, as comparing records one-by-one is computationally intensive and time-consuming, especially with millions of records.
Innovation Solution
The method involves generating clusters of records, identifying duplicate records within these clusters, merging duplicate sets based on common records, and removing duplicates to efficiently identify and remove duplicate records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If records are compared one-by-one to identify duplicates, then duplicate detection accuracy is improved, but processing time and computational intensity increase significantly
Solution Approach 1:
The patent divides the large set of records into multiple clusters based on common attributes (e.g., last name, first name, account name). Each cluster contains records that are potential duplicates of each other. By segmenting the comparison task into smaller cluster-level comparisons rather than comparing all records against all other records, the system maintains duplicate detection accuracy while significantly reducing processing time and computational intensity.
2Reliability
If all records are processed to ensure complete duplicate identification, then duplicate detection completeness is improved, but computational intensity increases significantly
Solution Approach 1:
The patent segments the record set into clusters based on matching key attributes. By processing each cluster separately and then merging results, the system ensures that all duplicates are identified (completeness) while avoiding the computational intensity of comparing every record against every other record. The clustering approach processes only relevant record subsets.
Solution Approach 2:
The patent merges duplicate sets identified from different clusters by finding common records between sets. This merging process ensures that duplicates spanning multiple clusters are correctly identified and consolidated, maintaining detection completeness while using efficient set operations rather than exhaustive comparisons.
Data Source
AI summary
Some embodiments of the present invention include a system and method for removing duplicate records from a group of records in a database system. The method includes generating a first cluster of records from the group of records, generating a second cluster of records from the group of records, identifying sets of duplicate records in the first cluster of records, and identifying sets of duplicate records in the second cluster of records. The method also includes merging at least two sets of duplicate records associated with both the first cluster and the second cluster of records to form a merged set of duplicate records. The merging is performed based on the at least two sets of duplicate records having a common record. Duplicate records in the group of records may then be removed by removing duplicate records from the merged set of duplicate records.