Genealogical Entity Resolution with Weighted Field Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing entity-resolution approaches in genealogical databases face challenges such as high computation costs, resource inefficiency, difficulty in updating, subjective rule-based algorithms, erroneous clustering, and lack of internal consistency, particularly when dealing with large-scale datasets and complex historical records.
Innovation Solution
A hybrid machine-learned, rule-based implementation that classifies entities as SAME, PLAUSIBLE, or DIFFERENT by independently scoring and combining field importance, using machine-learned weights to generate combinatorial probability scores, reducing computational overhead and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional entity-resolution approaches are used in genealogical databases, then existing methods can be applied, but computation costs are high and resource efficiency is poor
Solution Approach 1:
The entity-resolution process is divided into multiple independent scoring stages: name score, date score, location score, and relationship score. Each stage evaluates specific attributes separately using predefined rules and machine-learned models, then combines results to produce final match probability. This segmentation reduces computational complexity by avoiding exhaustive comparison of all attributes simultaneously.
Solution Approach 2:
The system pre-calculates and stores probability thresholds and scoring rules before actual entity-resolution operations. Machine-learned models are trained in advance on historical data to determine optimal weighting of different score components. This preliminary preparation eliminates the need for complex real-time calculations during entity-resolution operations.
2Measurement precision
If comprehensive comparison of all data fields is performed, then resolution accuracy improves, but processing time increases
Solution Approach 1:
The comparison process is segmented into independent attribute evaluations (name, date, location, relationships), each with its own scoring function. This allows the system to evaluate only relevant attributes for each record pair without processing all possible fields, reducing processing time while maintaining accuracy through focused comparison of critical identifiers.
Solution Approach 2:
Different scoring functions are applied to different attributes based on their local importance and characteristics. Name matching uses one set of rules, date matching uses another, and relationship matching uses a third. This localized approach allows optimized processing of each attribute type without the overhead of a uniform comprehensive comparison system.
3Ease of manufacture
If rule-based algorithms are used for entity resolution, then implementation is straightforward, but the algorithms become subjective and lack internal consistency
Solution Approach 1:
The system merges rule-based scoring functions with machine-learned probability models into a unified entity-resolution framework. The machine-learned model processes the outputs of multiple rule-based scorers and combines them using statistically optimal weighting, creating a hybrid system that maintains rule-based transparency while adding machine-learned consistency and objectivity through data-driven parameter optimization.
4Quantity of substance
If existing entity-resolution methods are applied to large-scale datasets, then processing is performed, but erroneous clustering occurs
Solution Approach 1:
The clustering process is segmented into multiple independent verification stages, each checking different aspects of potential matches. The system requires consistent results across multiple scoring dimensions (name, date, location, relationships) to confirm a match, preventing erroneous clustering that occurs in single-stage approaches when processing large datasets.
Solution Approach 2:
The machine-learned probability model provides feedback on the reliability of potential matches by calculating confidence scores based on the distribution of match probabilities observed in training data. This feedback mechanism allows the system to identify and reject false positives, improving clustering accuracy even as dataset size increases.
Data Source
AI summary
Systems, methods, and computer-program products for entity resolution are disclosed. Entity resolution embodiments include receiving tree data from each of a pair of entities, extracting and/or aggregating feature vectors or metric functions therefrom, and generating similarity scores between the pair of entities. The similarity scores may be weighted using machine-learned weights. The weighted similarity scores are used to generate a combinatorial probability score accounting for combined likelihoods of field values between the pair of entities. A classification of the pair of entities is performed based on the combinatorial probability score, with a genealogical database modified based on the classification.