Graph-Based Entity Deduplication with Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database deduplication systems fail to detect duplicate records in real-time and do not effectively utilize machine learning models for comparison.
Innovation Solution
A method involving generating record blocks, creating record pairs, calculating features, and inputting them into a machine learning model to predict probability scores for deduplication, with results stored as links in a graph for efficient duplicate detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional duplication search systems store duplications in a graph, then duplicate records can be identified, but real-time detection is not achieved
Solution Approach 1:
The system pre-calculates and stores probability scores for all record pairs in a graph structure before actual queries. This preliminary computation allows O(1) retrieval time during real-time operations, as the heavy computational work is done in advance and results are cached for immediate access.
Solution Approach 2:
The system continuously updates and maintains the probability score graph as new records are added or existing records are modified. This continuous maintenance ensures that the duplicate detection capability remains current and accurate without requiring full re-computation, enabling real-time detection while preserving accuracy.
2Productivity
If systems search records with a common token and assign scores, then record comparisons can be performed, but machine learning models are not utilized for accurate prediction
Solution Approach 1:
The system replaces traditional mechanical token-matching algorithms with a machine learning model that learns optimal feature combinations and weighting from training data. This substitution enables more accurate duplicate detection by capturing complex patterns that simple token matching cannot identify, while still maintaining computational efficiency through the pre-computed graph structure.
3Reliability
If all record pairs are compared to detect duplicates, then comprehensive duplicate detection is achieved, but search space becomes excessively large
Solution Approach 1:
The system segments the search space by organizing records into blocks and creating record pairs only within relevant blocks rather than comparing all possible pairs. This segmentation dramatically reduces the number of comparisons needed while maintaining detection completeness through the graph structure that connects related records across blocks.
Solution Approach 2:
The system changes the parameter space by transforming raw record attributes into a fixed set of normalized features that are fed into the machine learning model. This parameter transformation enables efficient computation of similarity scores while preserving the ability to detect all relevant duplicates through the learned feature relationships.
Data Source
AI summary
A computer-implemented method for deduplicating records includes generating a block comprising of a group of records. The method also includes creating one or more record pairs from the block, and calculating one or more features based on one or more fields within the one or more record pairs. The method further includes inputting the one or more features into a machine language trained model to predict a probability score. The probability score indicates whether two records are duplicates. The method also includes storing the probability score as links between two vertices in a graph, and displaying one or more duplicate records by querying the graph.