Entity Identity Matching Using Machine Learning Confidence Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Supply chain databases often contain redundant entries for the same company due to variations in naming conventions, typographical errors, and differences in address formats, leading to inefficiencies in data management.
Innovation Solution
A computer-implemented method and system that groups entity identifiers into pairs, uses similarity algorithms to determine potential matches, and employs a machine learning model to classify these pairs based on confidence scores, identifying entities that refer to the same company.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple entity identifiers are stored in the database, then the database can capture various naming conventions and variations, but redundant entries for the same company accumulate
Solution Approach 1:
The system performs preliminary entity resolution by grouping entity identifiers into pairs and calculating similarity scores before final database storage. This advance processing identifies potential duplicates early in the data ingestion pipeline, preventing redundant entries from being committed to the database in the first place
Solution Approach 2:
The machine learning model provides feedback loops that continuously learn from resolved entity pairs and improve matching accuracy over time. The system uses confidence scores to iteratively refine which entities are considered duplicates, adapting to new naming patterns and variations while reducing false positives
2Measurement precision
If similarity algorithms are applied to all entity pairs, then potential matches are identified, but computational resources are consumed
Solution Approach 1:
The system segments the entity matching process into distinct stages: initial grouping of entity identifiers into pairs, filtering through similarity algorithms to identify potential matches, and final classification by the machine learning model. This segmentation allows computational resources to be focused on promising candidates rather than exhaustively comparing all possible pairs
Solution Approach 2:
The system applies similarity algorithms selectively to entity pairs that meet certain criteria rather than all possible pairs. By using confidence thresholds and progressive filtering, the system performs partial action on the most likely candidates, achieving high accuracy while minimizing unnecessary computational expenditure
3Reliability
If a machine learning model is used to classify entity pairs, then confidence scores are generated for accurate matching, but system complexity increases
Solution Approach 1:
The machine learning model acts as an intermediary layer between the similarity algorithm results and the final entity resolution decision. It takes similarity scores and additional entity attributes as input, processes them through learned patterns, and outputs confidence scores that guide the matching decision, bridging the gap between simple string comparison and complex entity identity determination
4Measurement precision
If entity identifiers are grouped into pairs for comparison, then matching accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary grouping of entity identifiers into pairs based on basic criteria before applying complex similarity algorithms. This pre-filtering step reduces the total number of pairs that require detailed comparison, achieving accurate matching while minimizing processing time by eliminating obviously non-matching pairs early in the pipeline
Data Source
AI summary
A computer-implemented method for entity resolution is provided. The method comprises receiving a number of different entity identifiers and grouping the entity identifiers into a number of entity pairs. The entity pairs are fed into a match generator filter that determines a similarity score for each entity pair according to a number of similarity algorithms. Potentially matching entity pairs comprising a subset of the entity pairs that have similarity scores above a first specified threshold are then fed into a machine learning model that determines a confidence score for each potentially matching entity pair. The machine learning model identifies matched entities that comprise a subset of the potentially matching entity pairs that have confidence scores above a second specified threshold.