Entity Resolution System for Cross-List Record Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing filtering functions struggle to associate records across multiple lists without identical fields, especially in large datasets, leading to inefficiencies in identifying and isolating relevant records, as they rely on identical fields for comparison, which is not feasible when records originate from different sources and have varying field structures.
Innovation Solution
The system employs an entity resolution system that processes records from multiple lists, applies cleaning and blocking functions to normalize data, and uses pair generation and evaluation to associate records based on similar fields, reducing unnecessary comparisons and identifying related records without identical fields.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If filtering functions rely on identical fields across records, then record comparison and isolation can be achieved, but the system fails when lists have no identical fields or only a small number of identical fields
Solution Approach 1:
The system transforms the comparison parameter from requiring identical field names to using field values after normalization. Cleaning functions standardize formats (e.g., date formats, phone numbers), and blocking functions create comparison keys from potentially different field structures, allowing records to be matched based on equivalent data rather than identical field names.
Solution Approach 2:
The patent introduces intermediary structures including normalization rules, blocking functions, and scoring mechanisms that mediate between disparate field structures. These intermediaries translate different field representations into comparable forms, enabling record association even when source lists have completely different field names or structures.
2Measurement precision
If traditional filtering functions are used to compare records, then records with identical fields can be isolated, but comparison overhead increases significantly with large dataset sizes
Solution Approach 1:
The comparison process is segmented into distinct phases: cleaning functions prepare individual fields, blocking functions create preliminary groups from subsets of fields, and scoring functions evaluate pairs within blocks. This segmentation prevents full pairwise comparison across entire datasets, reducing computational overhead while maintaining matching accuracy.
Solution Approach 2:
The system performs preliminary actions through cleaning and blocking functions before the actual comparison. Cleaning normalizes data in advance, and blocking pre-groups records based on key fields, so that the expensive comparison operation only needs to evaluate a small subset of candidate pairs rather than all possible pairs in large datasets.
Data Source
AI summary
Computer implemented systems and methods are disclosed for associating records across lists, wherein the lists include a plurality of records and the plurality of records is associated with a respective entity. In accordance with some embodiments, the systems and methods further comprise grouping one or more records from a first list into a first group based on fields of the records in the first list, grouping one or more records from a second list into a second group based on fields of the records in the second list, pairing a record from the first group with a record from the second group, assessing each pair of records based on an evaluation of the respective pair according to fields of the pair, and associating records from the first group and records of the second group with an entity based on the assessment.