Supervised Entity Resolution Using Verified Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems using supervised machine learning face challenges in accurately training models when subject matter experts can only verify entire clusters, not pair-wise training labels, leading to biased training and inaccurate results.
Innovation Solution
The system learns effectively from verified clusters, allowing for rapid convergence to high-accuracy deduplication by using verified clusters as the primary source of training data, either alone or in combination with other training pairs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised machine learning is trained using pair-wise labeled examples, then the model can learn deduplication patterns, but subject matter experts cannot effectively verify accuracy since they can only judge entire clusters rather than individual pairs
Solution Approach 1:
The patent segments the verification task from the training task. Instead of requiring experts to verify individual training pairs, the system generates training pairs automatically from clusters that experts verify. This segmentation allows experts to work at the cluster level while the system handles pair-level training data generation.
Solution Approach 2:
The patent introduces an intermediary process that converts verified clusters into training pairs. This intermediary layer (the training data generation mechanism) bridges the gap between cluster-level expert verification and pair-level model training, allowing both to coexist without direct conflict.
2Ease of operation
If training labels are derived from verified clusters, then expert verification is simplified to cluster level, but this leads to biased training and inaccurate machine learning results
Solution Approach 1:
The patent applies dynamics by making the training pair generation process adaptive rather than static. The system dynamically generates diverse training pairs from verified clusters, including both positive pairs (within-cluster) and negative pairs (between-cluster), ensuring balanced and unbiased training data that maintains model accuracy while preserving verification simplicity.
3Quantity of substance
If more verified clusters are collected to improve training data quantity, then model training becomes more comprehensive, but the time and resources required for expert verification increase
Solution Approach 1:
The patent applies preliminary action by pre-processing verified clusters into training pairs in advance. This preliminary transformation allows the system to generate large quantities of training data from a relatively small number of verified clusters, reducing the need for experts to verify a large number of individual clusters while still achieving comprehensive model training.
4Measurement precision
If pair-wise training labels are used, then the machine learning model can be trained, but contradictory training labels may arise leading to biased training
Solution Approach 1:
The patent implements feedback by using the cluster verification results as ground truth to guide training pair generation. The system feeds back the expert-verified cluster structures into the training data generation process, ensuring that all generated training pairs are consistent with verified cluster assignments and eliminating contradictory labels.
Data Source
AI summary
Record clustering is performed by learning from verified clusters which are used as the source of training data in a deduplication workflow utilizing supervised machine learning.