Entity Matching Inference Optimization via Pre-Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Highly accurate machine learning models are complex, leading to longer training and inference times, as well as increased resource consumption, making them impractical for production use, especially in cloud environments with numerous simultaneous inference calls.
Innovation Solution
A machine learning platform that filters out unmatchable entity pairs prior to inference using a classifier to reduce the input data, optimizing inference time and conserving resources by processing a subset of entities, which includes identifying features through a trained classifier based on correlations and domain knowledge to predict matching pairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a highly accurate ML model is used for entity matching, then matching accuracy is improved, but inference time increases and the model becomes impractical for production use
Solution Approach 1:
The patent segments the entity matching process into two distinct stages: (1) a filtering stage using a simple classifier to identify candidate matches, and (2) a verification stage using the complex accurate ML model. This segmentation allows the system to use the accurate model only on a small subset of candidates rather than all entity pairs, thereby maintaining accuracy while reducing inference time.
Solution Approach 2:
The patent applies preliminary action by using a fast classifier to pre-filter entity pairs before they are processed by the accurate ML model. The classifier performs preliminary identification of potential matches based on simple features, eliminating clearly non-matching pairs beforehand. This preliminary filtering step reduces the workload of the accurate model and enables production-use inference times.
2Measurement precision
If a complex ML model is used for entity matching, then matching accuracy is improved, but resource consumption (computing power, memory) increases
Solution Approach 1:
The patent segments the processing workload by dividing entities into different processing paths: a fast classifier handles the majority of entity pairs with minimal resource consumption, while the resource-intensive accurate ML model is invoked only for candidate matches identified by the classifier. This segmentation dramatically reduces overall resource consumption while preserving accuracy for relevant matches.
Solution Approach 2:
The patent applies partial action by using the accurate ML model selectively only when necessary (for candidate matches) rather than applying it universally to all entity pairs. The simple classifier handles the partial majority of cases, and the accurate model is applied partially only to the subset of candidates that require verification, optimizing resource utilization.
Data Source
AI summary
Methods, systems, and computer-readable storage media for receiving input data including a set of entities of a first type and a set of entities of a second type, providing a set of features based on entities of the first type, the set of features including features expected to be included in entities of the second type, filtering entities of the second type based on the set of features to provide a sub-set of entities of the second type, and generating an output by processing the set of entities of the first type and the sub-set of entities of the second type through a ML model, the output comprising a set of matching pairs, each matching pair in the set of matching pairs comprising an entity of the set of entities of the first type and at least one entity of the sub-set of entities of the second type.