Greedy Entity Matching Inference With Threshold-Based Pair Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional entity matching methods using ML models require excessive computational resources and time due to redundant comparisons, especially when dealing with large datasets, leading to exponential scaling issues.
Innovation Solution
Implement greedy inference techniques that set probability thresholds based on ML model accuracy, selectively removing entities from further processing if their match probability meets or exceeds the threshold, using a global dictionary to track matched entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each entity in the query set is compared to all entities in the target set to obtain match probabilities, then comprehensive matching accuracy is improved, but computational costs and resource consumption increase exponentially
Solution Approach 1:
The patent applies preliminary action by performing a filtering step before the main comparison operation. Specifically, it pre-identifies and removes entities that cannot possibly match based on simple key comparison, eliminating them from the target set before the ML model performs probability calculations. This preliminary filtering reduces the number of entities that need expensive probabilistic comparisons, thereby lowering computational complexity while preserving matching accuracy for relevant entities.
Solution Approach 2:
The patent segments the entity matching process into two distinct stages: (1) a preliminary filtering stage that uses simple key comparison to identify and remove non-matching entities, and (2) a main inference stage that applies the ML model only to the reduced set of potential matches. This segmentation allows the system to handle large datasets efficiently by dividing the computationally intensive task into a cheap preliminary step and a focused expensive step.
2Reliability
If traditional inference processes all entity pairs through the ML model, then complete inference results are obtained, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent extracts and removes entities from the processing pipeline that have already been determined to match based on simple key comparison. By taking out these already-identified matches from the target set, the system eliminates the need to process them further through the time-consuming ML model inference, thereby reducing processing time while maintaining inference completeness for all relevant entities.
Solution Approach 2:
The patent applies partial action by performing complete ML model inference only on the subset of entity pairs that remain after preliminary filtering, rather than on all possible pairs. This partial processing approach is sufficient because the filtering step has already identified and removed entities that would not contribute to meaningful matches, making the partial inference both time-efficient and complete for relevant cases.
3Adaptability or versatility
If the number of query entities and target entities increases, then the scope of matching tasks is improved, but computational costs increase exponentially
Solution Approach 1:
The patent applies preliminary action by performing a scaling-friendly filtering operation that reduces the effective size of the target set before the main matching process. This preliminary key-based filtering scales linearly with dataset size and can be performed efficiently even on large datasets, creating a reduced problem size that the ML model can handle with significantly lower computational resource consumption while maintaining matching scope.
Solution Approach 2:
The patent changes the parameter of entity set size by dynamically reducing the target set based on the query entities being processed. Instead of maintaining a fixed large target set, the system adapts the target set size by removing non-matching entities through preliminary filtering, thereby changing the effective parameter of computational workload to match the actual matching needs and reduce resource consumption proportionally.
Data Source
AI summary
Methods, systems, and computer-readable storage media for determining a set of potential probability thresholds based on a set of inference results provided by processing testing data through the ML model, for each potential probability threshold in the set of potential probability thresholds, determining an accuracy, selecting a probability threshold from the set of potential probability thresholds, processing an inference job including sets of entity pairs through the ML model to assign a label to each entity pair in the sets of entity pairs, each label being associated with a probability and including a type of multiple types, and for each entity pair having a label of one or more specified types, selectively removing an entity of the entity pair from further processing of the inference job by the ML model based on whether the probability associated with the label meets or exceeds the probability threshold.