Travel Data Deduplication via ML Similarity Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In travel and transportation data processing, duplicate records of passengers are often not identified due to minor changes in field content, leading to human error and inefficiencies in data deduplication, where records of the same individual are not correctly merged or deleted.
Innovation Solution
A method and system utilizing a similarity model, potentially a machine learning model, to compare multi-field records and produce a probability of duplication, automatically merging records with high probability, queuing those with medium probability for manual review, and omitting those with low probability, with the model trained on correlated pairs and refined by manual feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data deduplication methods are used to compare records, then exact duplicate records can be identified, but records with minor variations in field content (such as different spellings or formats of the same name) are incorrectly treated as unique records
Solution Approach 1:
The patent introduces a similarity model as an intermediary component between the data records and the deduplication decision. This model receives record pairs, processes them through trained similarity metrics, and outputs a probability score indicating whether the records refer to the same individual. The similarity model acts as a mediator that handles the complexity of comparing varied field contents without requiring direct complex comparison logic in the deduplication system.
Solution Approach 2:
The patent replaces traditional mechanical string-matching and exact comparison methods with a machine learning-based similarity model. Instead of using straightforward character-by-character comparison algorithms, the system substitutes these with neural network-based similarity computation that can understand semantic equivalence between different representations of the same data (e.g., different spellings of names, varied date formats).
2Measurement precision
If manual review of all record pairs is performed to ensure accurate deduplication, then deduplication accuracy can be maintained, but processing time and operational efficiency decrease significantly
Solution Approach 1:
The patent applies partial automation where not all record pairs require the same level of processing. The similarity model provides a probability score for each pair, allowing the system to automatically process high-confidence matches (automatically merging records with high probability scores) while reserving manual review only for ambiguous cases (medium probability scores). This partial automation approach maintains high accuracy for clear-cut duplicates while significantly improving overall processing efficiency by avoiding manual review of all records.
Solution Approach 2:
The system enables self-service deduplication where the similarity model autonomously makes deduplication decisions for the majority of record pairs without human intervention. The model processes records independently, computes similarity probabilities, and triggers automated merging actions without requiring operator involvement. Manual review is reserved only for edge cases where the model's confidence is insufficient, allowing the system to serve itself for routine deduplication tasks while maintaining human oversight for complex situations.
3Productivity
If automated merging of all high-probability duplicate records is implemented, then processing speed increases, but the complexity of managing different probability thresholds and manual review queues increases
Solution Approach 1:
The patent segments the deduplication process into distinct handling paths based on probability thresholds. Record pairs are divided into three categories: high probability (automatically merge), medium probability (queue for manual review), and low probability (reject). This segmentation creates simple, well-defined workflows for each category, avoiding the complexity of managing a single unified process with multiple overlapping decision criteria. Each segment has its own straightforward processing logic, making the overall system easier to manage despite the multi-threshold approach.
Data Source
AI summary
A method, system and computer program product for data deduplication in a travel and transportation data processing system includes loading into memory for comparison from a database of a multiplicity of multi-field records, a pair of two different multi-field records and submitting the pair to a similarity model that correlates a companion passenger with a specified individual so as to indicate a probability of duplication whenever the companion passenger appears in a pair of records submitted to the model for comparison. Thereafter, if the model produces a high probability, the pair is automatically merged into a single record without manual intervention, but if the model produces a medium probability, the pair is placed in a queue pending manual intervention and manual merging, and if the similarity model produces a low probability, the pair is omitted from consideration of merging. Finally, the process repeats for each other pair of different multi-field records in the database.