Machine-Learning Entity Resolution with Multi-Pass Blocking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Entity resolution across large and inconsistent datasets is challenging due to data size, quality, and computational complexity, leading to incorrect or missed matches, which is critical for functionalities like customer data platforms, CRM systems, and fraud detection.
Innovation Solution
A machine-learning based entity resolution system that creates a unified view of entities by generating an entity graph and index, using multi-pass blocking, distributed clustering, and edge pruning to accurately connect and match entities across datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional entity resolution methods are used on large datasets, then completeness of entity matching is improved, but computational complexity and processing time increase significantly
Solution Approach 1:
The patent segments the entity resolution process into multiple passes with blocking techniques that divide large datasets into smaller, manageable blocks. Each pass processes specific subsets of data, reducing the overall computational complexity while maintaining matching accuracy through systematic segmentation of the resolution task.
Solution Approach 2:
The patent applies preliminary blocking and filtering actions before full entity resolution. By pre-processing data to identify and group potential matches using blocking keys, the system reduces the search space for subsequent resolution passes, thereby lowering computational complexity while preserving matching completeness.
2Reliability
If traditional entity resolution methods are used on large datasets, then completeness of entity matching is improved, but processing time increases significantly
Solution Approach 1:
The patent segments the entity resolution process into multiple passes with blocking techniques that divide large datasets into smaller, manageable blocks. Each pass processes specific subsets of data, reducing the overall computational complexity while maintaining matching accuracy through systematic segmentation of the resolution task.
Solution Approach 2:
The patent applies preliminary blocking and filtering actions before full entity resolution. By pre-processing data to identify and group potential matches using blocking keys, the system reduces the search space for subsequent resolution passes, thereby lowering computational complexity while preserving matching completeness.
3Productivity
If data quality is not addressed, then processing speed is maintained, but entity matching accuracy deteriorates due to inconsistencies
Solution Approach 1:
The patent implements dynamic data quality assessment and adaptation throughout the entity resolution process. The system adjusts its processing strategies based on detected data quality issues, dynamically modifying blocking keys, similarity thresholds, and resolution approaches to maintain accuracy while preserving processing efficiency.
Solution Approach 2:
The patent changes processing parameters adaptively based on data quality characteristics. By modifying similarity thresholds, blocking strategies, and matching criteria according to detected data inconsistencies, the system maintains high entity matching accuracy without sacrificing processing speed through rigid parameter settings.
Data Source
AI summary
An entity resolution method and system may obtain historical entity data from one or more databases to generate an entity matrix based on a clustering of the historical entity data. The entity resolution method and system may partition the entity matrix into one or more disjointed groups of records based on multi-pass blocking of the attributes associated with the one or more entities and cluster the one or more disjointed groups of records based on one or more similarity metrics between each of the one or more disjointed groups of records. The entity resolution method and system may generate an entity graph for the one or more entities based on the clustering and may create an entity index for an entity based on the entity graph. Ultimately, the entity resolution method and system resolves an entity query of a requestor as matching the entity.