Entity Resolution Match Vectors for Scalable Record Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional entity resolution systems face challenges with n-squared computational complexity, reliance on blocking keys that may not provide fine-grained partitioning, and lack of ground truth labels, leading to inefficiencies and inaccuracies in matching records.
Innovation Solution
A novel approach that constructs Boolean match vectors from exact-match conditions on record attributes, uses self-join and group-by-count operations to count occurrences, and trains an entity resolution model based on match vectors without relying on blocking keys, employing a supervised or unsupervised mixture model to determine record pair merges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional blocking key approach is used to group similar records, then the dataset is divided into smaller subsets for comparison, but the computational complexity still scales as n-squared and may miss matches due to insufficient partitioning granularity
Solution Approach 1:
The patent segments the entity resolution problem by dividing records into clusters based on vector similarity rather than using traditional blocking keys. This segmentation approach groups records by their overall vector proximity in n-dimensional space, creating more meaningful partitions that reduce the number of comparisons needed while maintaining match accuracy.
Solution Approach 2:
The patent transforms the entity resolution approach by changing from discrete blocking key categories to continuous vector space parameters. Records are represented as vectors in n-dimensional space, and similarity is measured using continuous distance metrics rather than discrete key matching, enabling more flexible and accurate partitioning.
2Measurement precision
If multiple blocking keys are used to improve match accuracy, then more attributes must be compared, but the computational complexity increases and scaling becomes insufficient
Solution Approach 1:
The patent merges multiple attribute comparisons into a single unified vector representation. Instead of comparing records across multiple blocking keys separately, all attributes are combined into n-dimensional vectors, and a single similarity calculation captures the overall match quality, reducing computational overhead while maintaining or improving accuracy.
Solution Approach 2:
The vector-based approach serves multiple functions simultaneously: it captures all attribute similarities, enables clustering at various granularities, and provides a unified similarity metric. This universal representation eliminates the need for multiple specialized blocking key comparisons.
3Reliability
If all possible record pairs are compared to ensure complete match detection, then match accuracy is maximized, but the computational time and resource requirements become unmanageable for large datasets
Solution Approach 1:
The patent performs preliminary clustering of records into groups based on vector similarity before conducting detailed pair-wise comparisons. This preliminary action identifies likely match candidates and groups them together, so that subsequent detailed comparisons are only performed within these pre-identified groups rather than across all possible pairs, significantly reducing computation time while maintaining completeness.
Data Source
AI summary
Systems and methods for performing entity resolution. In some aspects, the system obtains a plurality of attributes represented in a plurality of records from one or more sources. The system generates a plurality of match vectors based on the plurality of attributes. Each match vector includes a set of attributes that match between a pair of records and a remaining set of attributes that are not required to match between the pair of records. The system processes the plurality of match vectors using an entity resolution model trained to output a binary indicator regarding whether one or more pairs of records for a match vector be merged. The system merges the one or more pairs of records for each match vector of the plurality of match vectors having a corresponding binary indicator output from the entity resolution model that pairs of records for the match vector be merged.