Entity Resolution via Dynamic Threshold Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data filtering functions struggle to identify and resolve records from large lists without identical fields, especially when records originate from different sources, leading to inefficiencies as list sizes increase, making it difficult to isolate and aggregate common records effectively.
Innovation Solution
An entity resolution system that uses direct matching and advanced matching algorithms, including fuzzy keyword searching, fuzzy name searching, string comparators, and cascading thresholds, to analyze and score record pairs, dynamically adjusting threshold values based on match scores, enabling efficient identification of common records across multiple lists.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional filtering functions are used to identify common records, then identical fields must be present across records, but this limitation prevents effective resolution when lists have no identical fields or only a small number of identical fields
Solution Approach 1:
The system transforms the matching problem from exact field equality to similarity-based matching by changing the parameter from binary match/no-match to a continuous similarity score. This allows records to be matched based on degree of similarity rather than requiring identical fields, directly resolving the contradiction between adaptability and complexity.
Solution Approach 2:
The patent implements dynamic threshold adjustment where the matching threshold is not fixed but adapts based on the distribution of similarity scores in the data. This dynamic approach allows the system to handle varying degrees of field similarity across different datasets, enabling versatile matching while maintaining manageable complexity through adaptive behavior.
2Adaptability or versatility
If exact field matching is used to identify common records, then the process is simple, but it fails when records originate from different sources with varying field structures
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing similarity metrics for record pairs before the actual matching process. This preliminary computation of similarity scores allows the main matching operation to proceed more efficiently, reducing the time penalty associated with handling diverse record structures from different sources.
Solution Approach 2:
The patent replaces the mechanical approach of exact field-by-field comparison with a computational similarity assessment system. Instead of mechanically checking each field for equality, the system uses computational algorithms to assess overall record similarity, enabling efficient handling of records from different sources with varying field structures.
3Reliability
If comprehensive record comparison is performed across large lists, then all common records can be identified, but the computational burden increases significantly as list sizes grow to billions of records
Solution Approach 1:
The patent segments the large-scale matching problem into smaller, manageable units by dividing the comparison process into stages: initial filtering using key fields, then detailed similarity assessment only for candidate pairs. This segmentation maintains high accuracy for record resolution while dramatically improving productivity by avoiding exhaustive comparison of all record pairs in billion-record lists.
Solution Approach 2:
The system applies partial action by performing comprehensive similarity assessment only on a subset of candidate record pairs identified through initial filtering, rather than examining all possible pairs. This partial approach maintains sufficient reliability for identifying common records while achieving the productivity needed to handle billion-record lists efficiently.
Data Source
AI summary
Computer implemented systems and methods resolve data entries across multiple lists. The lists may include a plurality of records, wherein each record is associated with a respective entity. In accordance with some embodiments, the systems and methods further comprise identifying a direct field match between two lists, determining updated lists based on the remaining data entries, executing a comparison of the remaining data entries, determining a scoring metric based on the comparison, and determining whether the scoring metric exceeds a threshold. The systems and methods further comprise generating a data distribution curve based on the matched and unmatched data records and adjusting the threshold based on the data distribution curve for the next iteration of comparisons executed on the remaining unresolved entities.