Duplicate Entity Detection Using Inverted Indexes and ML
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for identifying and resolving duplicate entities in online networks are less accurate and comprehensive, leading to reduced quality and integrity of data, increased computational overhead, and decreased user trustworthiness due to the presence of duplicate entities in search results and recommendations.
Innovation Solution
A method using inverted indexes to identify pairs of entities with similar attributes, followed by machine learning models to generate confidence scores and classify duplicates, with user-annotated labels for training, to automatically detect and resolve duplicates by selecting a canonical entity and merging fields, thereby improving data integrity and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional techniques are used to identify duplicate entities, then the process is simpler, but the accuracy and comprehensiveness of duplicate detection deteriorates
Solution Approach 1:
The duplicate detection process is divided into multiple stages: initial filtering using inverted indexes, candidate pair generation, machine learning-based scoring, and final resolution. This segmentation allows each component to specialize in specific aspects of detection, improving overall accuracy while managing complexity through modular architecture.
Solution Approach 2:
Machine learning models serve as intermediaries between raw entity data and duplicate detection decisions. These models process features from entity pairs and generate confidence scores, acting as a sophisticated mediator that enhances detection accuracy without requiring direct complex rule-based comparisons between all entity pairs.
2Quantity of substance
If more entities are stored and processed, then the completeness of data is improved, but the computational overhead and resource consumption increases
Solution Approach 1:
The system extracts and processes only entity pairs that are likely to be duplicates, using inverted indexes to identify pairs with matching attributes. This extraction approach avoids processing all possible entity pairs, significantly reducing computational overhead while maintaining detection completeness for relevant cases.
Solution Approach 2:
The system applies partial action by focusing computational resources on candidate duplicate pairs rather than all entity pairs. Machine learning models evaluate only the most promising candidates, performing excessive action on a small subset to achieve overall system efficiency without compromising detection quality.
3Reliability
If duplicate entities are not resolved, then the data volume is maintained, but the quality and integrity of data deteriorates
Solution Approach 1:
The system implements feedback through confidence scoring and iterative refinement. Machine learning models generate confidence scores for candidate duplicate pairs, and the system can iteratively improve detection quality by learning from resolved cases. This feedback mechanism ensures high reliability in duplicate detection while systematically reducing the number of duplicate entities.
Solution Approach 2:
The system changes parameters by transforming entity attributes into feature representations suitable for machine learning processing. By converting raw data into standardized features and confidence scores, the system enables sophisticated duplicate detection while managing data volume through parameter-based filtering and scoring thresholds.
4Measurement precision
If manual review is used to verify duplicates, then the accuracy of resolution is improved, but the processing time and operational complexity increases
Solution Approach 1:
The system implements self-service through automated machine learning-based duplicate detection and confidence scoring. The majority of duplicate pairs are automatically identified and resolved without manual intervention, with the system serving itself by processing entity pairs through learned models. This reduces both processing time and operational complexity while maintaining high accuracy through algorithmic decision-making.
Data Source
AI summary
The disclosed embodiments provide a system that identifies duplicate entities. During operation, the system selects training data for a first machine learning model based on confidence scores representing likelihoods that pairs of entities in an online system are duplicates. Next, the system updates parameters of the first machine learning model based on features and labels in the training data. The system then identifies a first subset of additional pairs of the entities as duplicate entities based on scores generated by the first machine learning model from values of the features for the additional pairs and a first threshold associated with the scores. The system also determines a canonical entity in each of the duplicate entities based on additional features. Finally, the system updates content outputted in a user interface of the online system based on the identified first subset of the additional pairs.