Graph-Based Duplicate Record Clustering for Database Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face significant performance challenges when identifying and removing duplicate records, especially in large datasets, as existing methods like match rules become inefficient with millions of records, leading to increased processing time and resource consumption.
Innovation Solution
The method involves using a graph representation of records with nodes and edges to determine dense subsets, where records with the highest weight and satisfying a density threshold are identified as duplicates, allowing for efficient clustering and removal of duplicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If match rules are used to identify duplicate records in large datasets, then duplicate identification can be performed, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent segments the large dataset into multiple dense subsets using graph-based clustering. Instead of processing all records uniformly, the system divides them into smaller dense subsets where duplicates are more concentrated, enabling parallel processing and reducing overall processing time while maintaining identification accuracy
Solution Approach 2:
The patent changes the processing parameter from individual record comparison to dense subset identification. By computing density values and selecting subsets with density above a threshold, the system transforms the problem from O(n²) individual comparisons to identifying concentrated duplicate regions, significantly reducing processing time
2Reliability
If match rules are used to identify duplicate records in large datasets, then duplicate identification can be performed, but computing resources are consumed excessively
Solution Approach 1:
The patent segments the large dataset into multiple dense subsets using graph-based clustering. Instead of processing all records uniformly, the system divides them into smaller dense subsets where duplicates are more concentrated, enabling parallel processing and reducing overall processing time while maintaining identification accuracy
Solution Approach 2:
The patent changes the processing parameter from individual record comparison to dense subset identification. By computing density values and selecting subsets with density above a threshold, the system transforms the problem from O(n²) individual comparisons to identifying concentrated duplicate regions, significantly reducing processing time
3Productivity
If dense subset clustering is used to identify duplicate records, then processing time and resources are reduced, but the system complexity increases due to graph representation and density computation
Solution Approach 1:
The patent introduces a graph representation as an intermediary structure between the raw data and the duplicate identification process. The graph with nodes, edges, and density values serves as a mediator that simplifies the complex task of finding duplicates by providing a structured view of data relationships, making the overall system more manageable despite the added abstraction layer
Data Source
AI summary
Some embodiments of the present invention include a method for determining a dense subset from a group of records using a graphical representation of the group of records, the graphical representation having nodes and edges, a node associated with a record from the group of records, an edge connecting two nodes associated with two related records, wherein a node is associated with a weight corresponding to a number of edges connected to the node, wherein a record is added to the dense subset based on its associated node having a highest weight and a density that satisfies a density threshold, the density being based on the content of the dense subset, and wherein the content of the dense subset is to be processed as including duplicate records.