Entity Matching Filtering Using Learned Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems face inefficiencies in matching query entities to target entities, particularly in large datasets, due to the need to process all potential pairs, leading to high time complexity and resource usage, even with user-defined rules, which may not significantly reduce the number of inferred pairs and can compromise accuracy.
Innovation Solution
A machine learning system that uses learned embeddings to filter query-target entity pairs based on similarity scores, dynamically determining a filtering threshold to reduce the number of potential matches considered by the ML model, and storing filtered pairs in an efficient file structure to conserve memory and reduce inference time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all potential query-target entity pairs are processed by the ML model, then matching accuracy is maintained, but inference time and computational resources increase significantly
Solution Approach 1:
The system performs preliminary filtering by computing embeddings and similarity scores for all query-target entity pairs before feeding them to the ML model. A filtering threshold is determined in advance based on validation set performance, allowing the system to pre-identify and exclude low-similarity pairs from ML inference, thereby reducing computational time while maintaining accuracy.
Solution Approach 2:
The system extracts and removes irrelevant query-target entity pairs from the processing pipeline by filtering them out based on similarity scores computed from embeddings. Only pairs exceeding the filtering threshold are passed to the ML model, effectively taking out unnecessary computational work while preserving the accuracy of remaining matches.
2Productivity
If user-defined rules are used to filter entities, then some reduction in pairs is achieved, but the number of inferred pairs remains high and accuracy may be compromised
Solution Approach 1:
The system changes the filtering parameter from static user-defined rules to dynamic similarity thresholds based on learned embeddings. The filtering threshold is determined automatically from validation set performance, allowing adaptive filtering that maintains accuracy while significantly reducing the number of pairs processed compared to rigid rule-based approaches.
3Productivity
If embeddings and similarity filtering are applied, then the number of pairs processed is reduced, but computational overhead for embedding generation is introduced
Solution Approach 1:
The system performs embedding generation and similarity score computation as preliminary actions before ML inference. By pre-computing these representations and determining the filtering threshold in advance using validation data, the system efficiently filters pairs and reduces the computational burden on the ML model during actual inference, improving overall productivity.
Data Source
AI summary
Methods, systems, and computer-readable storage media for a machine learning (ML) system for matching a query entity to one or more target entities, the ML system that reducing a number of query-target entity pairs from consideration as potential matches during inference.