Entity Resolution Clustering for Cross-Source Record Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing filtering functions struggle to isolate relevant records from large lists lacking identical fields, especially when records originate from different sources, leading to inefficiencies as list sizes increase, making it difficult to identify and group related records effectively.
Innovation Solution
An entity resolution system that automatically analyzes large data structures, cleans and pairs records, scores their likelihood of relating to a common entity, and determines a canonical name for the entity, using techniques like Levenshtein distance and blocking models to efficiently group and identify records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If filtering functions are used to isolate records based on identical fields, then records can be isolated and aggregated to provide reports, but the system fails when records lack identical fields or have identical fields in only a small number of fields
Solution Approach 1:
The patent transforms the matching criterion from requiring identical field values to using similarity scoring based on multiple attributes. Records are compared across multiple fields (name, address, phone, etc.) and assigned similarity scores, allowing flexible matching even when no single field is identical. This resolves the contradiction by changing the parameter from exact match to similarity-based match.
Solution Approach 2:
The system creates a universal entity resolution framework that handles multiple data sources with different field structures. The entity resolution service can process records from various sources (telecommunications, finance, healthcare) using the same similarity-based approach, making the system adaptable to records with no identical fields across different sources.
2Productivity
If traditional filtering functions are used, then the process is simple for records with identical fields, but the system becomes inefficient as list sizes increase to billions of records
Solution Approach 1:
The patent segments the large-scale record comparison problem into manageable units by processing records in batches and using distributed computing. The entity resolution service divides the billion-record comparison task into smaller sub-tasks that can be parallelized, maintaining processing efficiency even as data volume increases to billions of records.
Solution Approach 2:
The system introduces an intermediary entity resolution service that acts as a mediator between raw data records and final analysis results. This service handles the complex similarity computation and record matching, freeing downstream systems from performing inefficient pairwise comparisons and maintaining overall productivity at scale.
3Measurement precision
If records are compared using multiple fields to improve accuracy, then more sophisticated filtering can be provided, but the complexity of the system increases
Solution Approach 1:
The entity resolution service automatically determines which fields to compare and how to weight them, eliminating the need for manual configuration of complex comparison rules. The system self-adjusts the matching criteria based on the data characteristics, providing high accuracy without requiring users to manage the underlying complexity of multi-field comparisons.
Data Source
AI summary
Computer implemented systems and methods are disclosed for automatically clustering and canonically identifying related data in various data structures. Data structures may include a plurality of records, wherein each record is associated with a respective entity. In accordance with some embodiments, the systems and methods further comprise identifying clusters of records associated with a respective entity by grouping the records into pairs, analyzing the respective pairs to determine a probability that both members of the pair relate to a common entity, and identifying a cluster of overlapping pairs to generate a collection of records relating to a common entity. Clusters may further be analyzed to determine canonical names or other properties for the respective entities by analyzing record fields and identifying similarities.