Entity Resolution via ML Scoring and Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems face challenges in efficiently evaluating and merging entity records due to duplicative information, which requires significant resource costs and yields difficult-to-understand results.
Innovation Solution
A computer-based method involving a processor that receives entity record sets, generates candidate pairs based on similarity, uses a scoring machine learning model to determine match scores, and merges records into entity records, updating an entity database efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional entity record evaluation methods are used to handle duplicative information, then comprehensive entity matching can be achieved, but resource consumption increases significantly and results become difficult to understand
Solution Approach 1:
The entity resolution process is segmented into distinct phases: candidate pair generation using blocking techniques, feature extraction, machine learning-based scoring, and cluster determination. This segmentation allows each phase to be optimized independently, reducing overall resource consumption while maintaining matching accuracy.
Solution Approach 2:
The system performs preliminary blocking to generate candidate pairs before full evaluation. By pre-filtering records that are likely to match based on blocking criteria, the system reduces the number of records requiring comprehensive evaluation, thereby reducing resource consumption while preserving matching accuracy.
2Reliability
If comprehensive entity record evaluation is performed on all records, then accurate entity resolution can be achieved, but processing time and computational resources increase
Solution Approach 1:
The evaluation process is divided into candidate pair generation, feature extraction, scoring, and clustering phases. Each phase processes only the necessary subset of records, reducing total processing time while maintaining resolution accuracy through systematic progression through evaluation stages.
Solution Approach 2:
Blocking techniques are applied preliminarily to identify candidate pairs before detailed evaluation. This preliminary action filters out non-matching records early, reducing the time required for comprehensive evaluation while preserving accuracy for potential matches.
3Measurement precision
If machine learning models are used to score candidate pairs, then match score accuracy improves, but computational complexity increases
Solution Approach 1:
The computational process is segmented into feature extraction and model scoring phases. By preparing features systematically before scoring and using efficient model architectures, the system achieves accurate match scores while managing computational complexity through structured processing.
Solution Approach 2:
Features are extracted and prepared preliminarily before being input to the machine learning scoring model. This preliminary feature engineering reduces the computational burden during the scoring phase while maintaining match score accuracy through comprehensive feature representation.
4Reliability
If all candidate pairs are evaluated in detail, then complete entity matching can be achieved, but resource costs and processing overhead increase
Solution Approach 1:
The evaluation process is segmented into candidate pair generation, detailed evaluation, and cluster determination phases. This segmentation enables the system to process large numbers of candidate pairs efficiently by focusing detailed evaluation only on pairs that pass initial filtering, maintaining matching completeness while improving processing efficiency.
Solution Approach 2:
Blocking and candidate pair generation are performed as preliminary actions to identify potential matches before detailed evaluation. This preliminary filtering reduces the number of pairs requiring resource-intensive detailed evaluation, improving processing efficiency while preserving matching completeness through systematic candidate identification.
Data Source
AI summary
In order to facilitate entity resolution, systems and methods include a processor receiving first records associated with one or more entities, and second records associated with the one or more entities. The processor generates candidate pairs based on a similarity between first entity data and second entity data. The processor generates features for each candidate pair based on similarity measures between the first entity record and the second entity record. The processor utilizes a scoring machine learning model to determine a match score for each candidate pair based on each feature. The processor determines clusters of candidate pairs based on the match score of each feature for each candidate pair. The processor merges records of candidate pairs of each cluster into a respective entity record. The processor determines an entity associated with each entity record and updates an entity database with the entity record.