Entity-Based Record Grouping Using Similarity Graph Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems face challenges in efficiently grouping data records associated with entities due to unclear associations between records, especially at a large scale, and the difficulty in determining whether records refer to the same or related entities.
Innovation Solution
The system employs a processor to identify pairs of data records, determine their similarity based on attributes, and cluster them according to unique entities, using a similarity function learned from training data and graph clustering techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data analytics systems gather information from millions of distributed repositories, then the quantity of information increases, but the difficulty of consuming information efficiently increases
Solution Approach 1:
The system segments the large volume of distributed information into discrete data records, each associated with specific entities. By breaking down the overwhelming information into manageable units with clear entity associations, the system enables efficient processing and consumption while maintaining the comprehensive quantity of gathered information
Solution Approach 2:
The system introduces an intermediary layer (the data analytics system with entity association mechanisms) between the distributed repositories and the information consumer. This intermediary gathers, processes, and organizes information from millions of repositories, presenting it in a consumable format through entity-based grouping and summary generation
2Measurement precision
If the system determines similarity values for all pairs of data records, then the accuracy of entity association improves, but the computational time and resources increase significantly
Solution Approach 1:
The system applies partial action by determining similarity values selectively rather than for all possible pairs. It focuses computational effort on pairs of data records that have potential entity associations based on initial filtering criteria, achieving sufficient accuracy for entity association without the prohibitive cost of exhaustive comparison
Solution Approach 2:
The system performs preliminary filtering and grouping of data records before computing similarity values. By pre-organizing records based on observable attributes and potential entity relationships, the system reduces the search space for similarity computations, maintaining accuracy while significantly reducing computational time
3Measurement precision
If the system clusters data records based on multiple attributes, then the accuracy of grouping records with the same entity improves, but the complexity of the clustering algorithm increases
Solution Approach 1:
The system segments the multiple attributes into hierarchical groups and processing stages. Rather than simultaneously analyzing all attributes together, it processes them in organized groups, reducing the computational complexity of the clustering algorithm while maintaining the ability to accurately group records based on multiple attributes
Solution Approach 2:
The system applies partial action by not requiring all attributes to be processed with equal depth for every record pair. It uses multiple attributes strategically, applying more rigorous analysis only where needed to resolve ambiguous associations, thereby maintaining grouping accuracy while managing algorithmic complexity
Data Source
AI summary
The present application relates to apparatus, systems, and methods for grouping data records based on entities referenced by the data records. The disclosed grouping mechanism can include determining a pair-wise similarity between a large number of data records, and clustering a subset of the data records based on their pair-wise similarity.