Adaptive Field-Level Matching for Database Record Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face inefficiencies in matching records due to the intensive resource consumption of deep matching processes, which often include false positives, necessitating a more accurate and resource-efficient method for field-level matching.
Innovation Solution
Implementing a machine-learning model for adaptive field-level matching that identifies features in record fields, calculates field match scores, and weighs these scores to determine record match scores, thereby reducing the need for human-configured algorithms and improving matching accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deep matching process is used to determine record matching, then matching accuracy is improved, but system resource consumption increases significantly
Solution Approach 1:
The matching process is divided into two distinct stages: shallow matching that uses a limited number of fields for initial candidate identification, and deep matching that applies intensive multi-field comparison only to the reduced candidate set. This segmentation allows the system to maintain high matching accuracy through deep matching while reducing overall resource consumption by limiting deep matching to only necessary cases.
Solution Approach 2:
The system applies deep matching selectively rather than universally - only to candidate records that pass the shallow matching threshold. This partial action approach ensures that the computationally intensive deep matching process is performed only where necessary to achieve accurate results, rather than being applied excessively to all possible record pairs.
2Use of energy by moving object
If a shallow matching process is used to reduce resource consumption, then system resource usage decreases, but false positive matching records increase
Solution Approach 1:
The two-stage matching process segments the evaluation into shallow matching for initial filtering and deep matching for final verification. This segmentation allows the system to tolerate some false positives in the shallow stage while maintaining overall reliability through the subsequent deep matching stage that eliminates false positives among the candidate records.
Solution Approach 2:
The shallow matching process acts as an intermediary stage between the initial large set of records and the final deep matching evaluation. It produces a intermediate candidate set that balances resource efficiency with maintaining sufficient matching reliability by filtering out clearly non-matching records before they reach the resource-intensive deep matching stage.
3Measurement precision
If multiple field comparisons are performed in deep matching, then matching accuracy is improved, but processing time increases
Solution Approach 1:
The multi-field comparison process is segmented into shallow matching (using fewer fields) and deep matching (using multiple fields). By segmenting the field comparison process and applying it at different stages, the system achieves high matching accuracy through multiple field comparisons while minimizing total processing time by limiting extensive field comparisons to only the candidate records that pass initial filtering.
4Measurement precision
If normalization and standardization processes are applied to record fields, then matching accuracy is improved, but device complexity increases
Solution Approach 1:
The normalization and standardization operations are segmented and applied specifically to the candidate records identified by shallow matching, rather than to all records in the database. This segmentation reduces the overall processing complexity while maintaining matching accuracy, as the intensive field-level normalization is performed only on the smaller candidate set that requires detailed evaluation.
Data Source
AI summary
Adaptive field-level matching is described. A system identifies first elements in a field of a prospective record for a database, and second elements in the field of a candidate record, in the database, for matching the prospective record. The system identifies features corresponding to any of the first elements that are identical to any of the second elements, any of the first elements that are absent from the second elements, and any of the second elements that are absent from the first elements. A machine-learning model uses the features to determine a field match score for the candidate record's field. Another machine-learning model weighs the field match score and weighs another field match score for another field of the candidate record to determine a record match score for the candidate record. If the record match score satisfies a threshold, the system identifies the candidate record as matching the prospective record.