Active Learning Cluster Selection for Entity Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for entity resolution in large data sets using supervised machine learning lack an effective way to combine active learning with cluster-based training, leading to inefficient convergence on high-accuracy models due to inadequate cluster selection and high computational costs.
Innovation Solution
A method and system for selecting clusters as training data in an active learning workflow, using approximate weighting and uncertainty metrics to rapidly converge on high-accuracy entity resolution, which is computationally efficient and parallelizable on commodity infrastructure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional supervised machine learning is used for entity resolution in large data sets, then model accuracy can be achieved, but the computational cost is high and convergence is slow
Solution Approach 1:
The patent applies preliminary action by pre-computing cluster centroids and uncertainty metrics before the main training process. Clusters are pre-formed from the large data set, and their uncertainty scores are calculated in advance, allowing the active learning workflow to efficiently select training clusters without performing these computationally intensive operations during each training iteration.
Solution Approach 2:
The patent segments the large data set into multiple clusters, each representing a subset of records with similar characteristics. By dividing the data into manageable clusters and computing uncertainty metrics for each cluster independently, the system can efficiently select representative training clusters without processing the entire data set at once, reducing computational overhead and improving convergence speed.
2Measurement precision
If more training clusters are used to improve model accuracy, then better accuracy is achieved, but computational costs increase
Solution Approach 1:
The patent changes the parameter of cluster selection by introducing uncertainty metrics as a selection criterion. Instead of using all clusters or random selection, the system computes uncertainty scores for each cluster and selectively trains on clusters with higher uncertainty values. This parameter change allows the system to achieve good model accuracy with fewer training clusters, reducing computational costs while maintaining effectiveness.
3Measurement precision
If exact cluster weighting methods are used to achieve accurate training cluster selection, then model accuracy improves, but computational complexity increases
Solution Approach 1:
The patent employs approximate uncertainty metrics that are computationally inexpensive to calculate, sacrificing some precision for significant gains in efficiency. These approximate metrics serve as disposable proxies for more complex exact methods, providing sufficient guidance for cluster selection without the prohibitive computational cost of exact weighting methods, especially in distributed computing environments.
4Ease of manufacture
If traditional cluster selection methods are used, then implementation is simple, but data efficiency is poor and more training clusters are required
Solution Approach 1:
The patent implements feedback by using uncertainty metrics to guide the selection of training clusters. The system computes uncertainty scores for clusters, uses these scores to select which clusters to train on, and then re-evaluates uncertainty after training to identify the next set of training clusters. This feedback loop ensures that each training iteration focuses on the most informative clusters, improving data efficiency and reducing the total number of training clusters needed compared to random or naive selection methods.
Data Source
AI summary
A collection of clusters are selected to be used in training in an active learning workflow when using clusters to train supervised entity resolution in data sets. A collection of records is provided wherein each record in the collection has a cluster membership. A collection of record pairs is also provided, each record pair containing two distinct records from the collection of records, and each record pair having a similarity score. A collection of clusters is generated with uncertainty from the collection of records and the collection of record pairs. A subset of the collection of clusters with uncertainty is then selected using weighted sampling, wherein a function of the cluster uncertainty is used as the weight in the weighted sampling. The subset of the collection of clusters with uncertainty is the collection of clusters for training in and active learning workflow when using clusters to train supervised entity resolution in data sets.