Unsupervised Transfer Learning for Intra-Cluster Entity Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms for similarity-based clustering often generate clusters with highly dissimilar records, making it challenging to reliably identify and separate duplicate or closely matching entities from large datasets, particularly in applications like database merging and item catalog organization.
Innovation Solution
The use of unsupervised transfer learning for attribute-level analysis, where an entity-pair similarity detection model is reused to automatically generate labels for attribute-value-pair similarity detection models, allowing for the identification and partitioning of divergent clusters, thereby improving cluster purity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing similarity-based clustering algorithms are used, then clustering can be performed on large datasets, but the clusters generated contain highly dissimilar records
Solution Approach 1:
The patent segments the clustering problem into multiple stages: initial clustering using entity-pair similarity detection, followed by intra-cluster analysis using attribute-level similarity detection. This segmentation allows each stage to focus on specific aspects of similarity, improving overall cluster purity while managing computational complexity.
Solution Approach 2:
The patent introduces attribute-value-pair similarity detection as an intermediary mechanism between initial clustering and final cluster validation. This intermediary layer analyzes specific attributes within clusters to identify and separate dissimilar records, thereby improving measurement precision without requiring complete re-clustering.
2Reliability
If manual annotation is used to improve clustering quality, then cluster purity increases, but computational resources and time increase significantly
Solution Approach 1:
The system performs self-service by automatically generating training data for attribute-level similarity detection using the initial clustering results. The entity-pair similarity detection model automatically labels attribute-value pairs without human intervention, enabling the system to improve its own clustering quality while maintaining high productivity.
Solution Approach 2:
The patent performs preliminary clustering using entity-pair similarity detection before conducting attribute-level analysis. This preliminary action creates initial clusters that serve as the basis for subsequent automated label generation, reducing the need for manual annotation while still achieving high cluster purity.
3Measurement precision
If more computational resources are dedicated to labeling and training, then model accuracy improves, but resource consumption increases
Solution Approach 1:
The patent applies partial action by focusing computational resources on analyzing only those clusters that require improvement, rather than processing all clusters uniformly. The attribute-level similarity detection is applied selectively to enhance measurement precision where needed while minimizing overall resource consumption.
Data Source
AI summary
Entity record pairs are extracted from a selected cluster of entity records. Attribute value pairs are obtained from the entity record pairs. Labels are assigned to the attribute value pairs based at least in part on entity-level similarity scores of the entity records from which the attribute value pairs were obtained. A machine learning model is trained, using a data set which includes at least some attribute value pairs to which the labels are assigned, to generate attribute similarity scores for pairs of attribute values.