Record Matching via Keyboard Distance and Probability Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional database search methods often return incomplete results due to variations in identifier representation and mistyped data, leading to inaccurate matching and fragmented files, as they fail to account for uncertainties and nuances in identifiers.
Innovation Solution
A record-matching computing system that generates identifier scores based on keyboard distance and probability distributions for numerical identifiers, and combines scores for overall matching, using machine learning models to determine matches and detect fragmented files by identifying connected components in a graph representation of database records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional exact matching methods are used for database searches, then search speed is maintained at acceptable levels, but matching accuracy deteriorates due to inability to handle variations and typos in identifiers
Solution Approach 1:
The matching process is segmented into multiple independent scoring components: keyboard distance score for numerical identifiers, probability distribution score for error analysis, and date identifier score for temporal matching. Each component evaluates a specific aspect of identifier similarity, and the results are combined to produce an overall matching score, enabling comprehensive accuracy without a single complex algorithm
Solution Approach 2:
The system changes the matching parameter from binary exact match to continuous similarity scoring. By introducing keyboard distance calculations and probability distribution-based error modeling, the system transforms identifier comparison into a graded assessment that accounts for typographical variations and data entry errors, improving accuracy while maintaining computational efficiency
2Reliability
If traditional exact matching methods are used, then database structure remains simple, but fragmented files increase due to duplicate records created from inaccurate matching
Solution Approach 1:
The system implements feedback through matched record detection, where the fuzzy matching algorithm identifies and merges duplicate records that would otherwise be created as separate entries. By continuously monitoring and correcting matching accuracy, the system prevents fragmentation and maintains data integrity, reducing the need for manual intervention
Solution Approach 2:
The system merges duplicate or near-duplicate records into single unified entries based on the overall matching score. By combining fragmented records that represent the same entity, the system reduces database size and eliminates redundancy, thereby improving data integrity and reducing storage requirements without losing information
3Measurement precision
If fuzzy matching with multiple score components is implemented, then matching accuracy improves, but computational complexity increases
Solution Approach 1:
The system applies partial action by calculating only the necessary score components based on the type of identifier being matched. For numerical identifiers, it computes keyboard distance and probability distribution scores; for date identifiers, it computes temporal distance scores. This selective computation approach ensures high accuracy where needed while minimizing unnecessary computational overhead, balancing precision with time efficiency
Data Source
AI summary
In some aspects, a record-matching computing system for matching records to facilitate database search and fragmented records detection is provided. The record-matching computing system is configured to receiving a query record and search in a data repository storing data records for a record that matches the query record. The record-matching computing system retrieves a reference record from the data records and generates multiple identifier scores. Each identifier score measures a degree of matching between the corresponding identifiers in the query record and the reference record. The record-matching computing system generates an overall matching score by combining at least two of the identifier scores and determines the reference record as a match to the query record based on the overall matching score exceeding a threshold value.