Entity Resolution Using Deterministic and Probabilistic Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Entity resolution between datasets, particularly between traditional and non-traditional datasets, is challenging due to the lack of strong identifying attributes in non-traditional datasets, leading to inefficiencies in pairwise matching and disambiguation processes.
Innovation Solution
A method and system that create deterministic and probabilistic models to compare datasets, using matching predicates, precedence rules, and cardinality rules to select candidate matches, and apply these models to traditional and non-traditional datasets to improve accuracy and recall in entity resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional pairwise matching is used for entity resolution, then the process is simple to implement, but accuracy deteriorates when datasets contain missing values or lack strong identifying attributes
Solution Approach 1:
The entity resolution process is segmented into distinct phases: deterministic model execution (applying precedence rules and matching predicates) followed by probabilistic model execution (applying cardinality rules). This segmentation allows each model type to address specific aspects of the matching problem, improving overall accuracy while maintaining manageable complexity through modular processing.
Solution Approach 2:
The deterministic model is executed first to perform preliminary filtering and candidate selection using precedence rules and matching predicates. This preliminary action reduces the candidate set before the probabilistic model applies cardinality rules, thereby improving efficiency and accuracy without requiring the full complexity of both models to run simultaneously.
2Measurement precision
If multiple algorithms and rules are applied to improve entity resolution accuracy, then accuracy and recall improve, but processing time and computational resources increase
Solution Approach 1:
The deterministic model with precedence rules performs preliminary filtering to eliminate obvious non-matches before the probabilistic model with cardinality rules processes the data. This preliminary action reduces the number of candidates requiring computationally intensive probabilistic analysis, thereby reducing overall processing time while maintaining high accuracy.
Solution Approach 2:
The system applies deterministic rules partially (only for initial filtering) and probabilistic rules selectively (only for remaining candidates). This partial application of computational methods optimizes the balance between accuracy and processing time by avoiding unnecessary computation on candidates that can be quickly eliminated.
3Productivity
If deterministic models with precedence rules are used, then processing efficiency improves, but ability to handle ambiguous cases deteriorates
Solution Approach 1:
The system merges deterministic model output (from precedence rules and matching predicates) with probabilistic model output (from cardinality rules). This combination allows the system to maintain high processing efficiency through deterministic filtering while simultaneously improving adaptability to ambiguous cases through probabilistic reasoning on the filtered candidates.
Solution Approach 2:
The deterministic model acts as an intermediary that filters and prioritizes candidates before they reach the probabilistic model. This intermediary processing maintains efficiency by handling clear-cut cases deterministically while allowing the probabilistic model to focus computational resources on ambiguous cases that require more sophisticated analysis.
4Measurement precision
If data steward disambiguation is performed manually, then decision accuracy improves, but processing speed and scalability deteriorate
Solution Approach 1:
The system implements self-service entity resolution through automated deterministic and probabilistic models that perform disambiguation without requiring manual data steward intervention for each case. The deterministic precedence rules and probabilistic cardinality rules work together to automatically resolve ambiguities, maintaining high processing speed and scalability while achieving accuracy comparable to manual disambiguation.
Data Source
AI summary
Described herein are methods, systems and computer program products for entity resolution. Entity resolution, also known as entity matching or record linkage, seeks to identify equivalent data objects between or among datasets. An example method includes creating a deterministic model by defining an entity to be resolved, selecting two datasets for comparison, defining matching predicates for attributes of the datasets to select a set of candidate matches, and defining a precedence rule for the candidate matches to select a subset of the candidate matches. The method includes running the deterministic model on the two datasets. Running the deterministic model includes applying the matching predicates and the precedence rule to data in the datasets that correspond to the attributes. The method also includes applying a cardinality rule to results of the running, and outputting the matching candidates for which the cardinality rule is satisfied.