Data Entity Matching with Feature Reduction and Candidate Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data integration techniques fail to account for discrepancies between data records for the same data entity, leading to duplicate records and inefficiencies in data entity matching across disparate data sources.
Innovation Solution
Implementing feature space reduction and feature engineering techniques to generate numeric vectors and data features, which are used to train a data entity matching model, enabling accurate matching despite record discrepancies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional exact matching techniques are used to match data entities across disparate data sources, then the matching process is simple and fast, but the accuracy deteriorates when data discrepancies such as typographical errors exist
Solution Approach 1:
The patent segments the matching process into multiple stages: initial exact matching, fuzzy matching for discrepancy handling, and machine learning-based verification. This segmentation allows the system to achieve high accuracy for records with discrepancies while maintaining simplicity for exact matches, resolving the contradiction between accuracy and complexity.
Solution Approach 2:
The patent changes the matching parameters dynamically based on data quality. For records with high similarity, exact matching parameters are used; for records with discrepancies, fuzzy matching parameters with tolerance thresholds are applied. This parameter adaptation enables the system to maintain simplicity where possible while achieving high accuracy when needed.
2Measurement precision
If fuzzy matching techniques are applied to handle data discrepancies, then the accuracy of matching improves, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies partial fuzzy matching only to specific fields that are likely to contain discrepancies (such as names and addresses) rather than all fields. This selective application reduces processing time while maintaining accuracy for the most critical matching criteria, resolving the contradiction between comprehensive accuracy and processing efficiency.
Solution Approach 2:
The patent performs preliminary filtering using exact matching on unique identifiers before applying fuzzy matching. This preliminary action eliminates the need for fuzzy matching in cases where exact matches are found, significantly reducing processing time while maintaining accuracy for records that require detailed verification.
3Measurement precision
If comprehensive feature engineering is performed on all candidate data objects, then the quality of match predictions improves, but the computational resources and processing time required increase
Solution Approach 1:
The patent applies different levels of feature engineering to different candidate data objects based on their similarity scores. High-similarity candidates receive comprehensive feature engineering, while lower-similarity candidates receive minimal or no feature engineering. This local quality approach maintains high prediction quality for likely matches while preserving processing throughput by avoiding unnecessary computation on unlikely candidates.
Solution Approach 2:
The patent performs feature engineering only on a subset of the most promising candidate matches rather than all candidates. This partial action approach focuses computational resources on the most critical matching decisions, maintaining high prediction quality for the most likely matches while preserving overall processing throughput by limiting extensive analysis to a manageable subset.
Data Source
AI summary
Embodiments of the present disclosure provide systems and methods for matching data entities across disparate datasets. One method may include generating a reference numeric vector for a reference data object and generating a plurality of match candidate numeric vectors for a plurality of match candidate data objects. The method may also include identifying a subset of match candidate data objects from the plurality of match candidate data objects and generating a plurality of data features based on a comparison between one or more reference object attributes corresponding to the reference data object and one or more matching object attributes corresponding to the subset of match candidate data objects. The method may also include identifying a match between the reference data object and a match candidate data object of the subset of match candidate data objects based on the plurality of data features.