Blocking Scheme Training Using Labeled and Unlabeled Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing record linkage methods face computational challenges with large data sets and perform poorly when trained solely on labeled data, generating too many candidate matches due to insufficient characterization of unlabeled data.
Innovation Solution
Training a blocking scheme using both labeled and unlabeled data to identify blocking attributes and comparison methods, maximizing true matches and minimizing non-matches by iteratively learning conjunctions and combining them using disjunctions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a blocking scheme is trained using only labeled data, then the training process is simpler and faster, but the blocking scheme performs poorly on unlabeled data by generating too many candidate matches
Solution Approach 1:
The patent applies preliminary action by pre-processing unlabeled data to generate pseudo-labels before training the blocking scheme. This preliminary labeling step enables the model to learn from unlabeled data without requiring manual annotation, thereby improving performance on unseen data while maintaining a manageable training process
Solution Approach 2:
The patent uses copying by creating synthetic labeled data from unlabeled data through pseudo-labeling. This allows the training process to leverage the large volume of unlabeled data by converting it into a format suitable for supervised learning, thus improving generalization to unseen data without proportionally increasing training complexity
2Reliability
If all pairs of records are compared to ensure complete linkage, then linkage accuracy is maximized, but computational cost becomes prohibitive for large data sets
Solution Approach 1:
The patent applies segmentation by dividing the record comparison task into two stages: first segmenting records into blocks based on blocking attributes, then only comparing records within the same block. This segmentation dramatically reduces the number of comparisons needed while maintaining linkage accuracy for the subset of records that require processing
Solution Approach 2:
The patent applies local quality by applying different levels of comparison strictness to different record pairs based on their blocking attributes. Records with similar blocking attributes are compared with higher precision, while records with dissimilar attributes are not compared, optimizing computational resources for the most promising matches
Data Source
AI summary
Some implementations disclosed herein provide techniques and arrangements to train a blocking scheme using both labeled data and unlabeled data. For example, training the blocking scheme may include iteratively: learning a conjunction, identifying first matches in the labeled data and the unlabeled data that are uncovered by the conjunction, and identifying second matches in the labeled data and the unlabeled data that are covered by the conjunction. The conjunction learned in each iteration may be combined using a disjunction. A search engine may use the search engine when searching for records that match an entity.