Entity-Based Record Clustering With Distributed Similarity Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently grouping large volumes of distributed data records associated with entities is complicated by the lack of clear identification of whether records refer to the same or related entities, due to variations in attribute formats and misspellings, making it difficult to create meaningful summaries.
Innovation Solution
A mechanism that learns a similarity function to cluster data records by analyzing attributes, uses domain-specific rules, and distributes computations across multiple devices to efficiently group data records, while pruning unnecessary comparisons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data records are grouped using traditional methods, then the process becomes manageable, but it cannot efficiently handle billions of distributed data records with variations in attribute formats and misspellings
Solution Approach 1:
The patent replaces traditional mechanical data processing methods with machine learning-based similarity computation. The system uses learned similarity functions and domain-specific rules to automatically identify entity relationships, substituting manual or rule-based mechanical approaches with intelligent algorithms that can handle billions of records while maintaining high accuracy in entity identification
Solution Approach 2:
The patent transforms the data processing approach by changing parameters from exact attribute matching to similarity-based matching. The system computes similarity values between data records using multiple attributes, allowing flexible matching that accounts for format variations and misspellings while maintaining identification accuracy across large-scale distributed data
2Measurement precision
If all pairs of data records are compared to determine similarity, then accurate clustering is achieved, but the computational burden becomes prohibitive for billions of records
Solution Approach 1:
The patent segments the computational task by dividing data records into multiple partitions distributed across different computing devices. Each device processes a subset of records independently, computing similarities only within its partition. This segmentation reduces the computational burden from O(n²) for all pairs to O(n²/k) for k partitions, enabling scalable processing of billions of records while maintaining clustering accuracy through coordinated aggregation of results
Solution Approach 2:
The patent implements partial action by computing similarities only for record pairs that meet certain criteria or belong to the same partition, rather than exhaustively comparing all possible pairs. The system uses domain-specific rules and learned models to identify promising candidate pairs for comparison, performing partial computations that are sufficient to achieve accurate clustering without the prohibitive cost of complete pairwise analysis
3Measurement precision
If domain-specific rules and learned similarity functions are used, then clustering accuracy improves, but the system complexity increases
Solution Approach 1:
The patent implements a universal clustering framework that can handle multiple entity types and domains through a single multi-functional system. The learned similarity functions and domain-specific rules are designed to be adaptable across different data types (e.g., businesses, locations, products) without requiring separate specialized systems. This universality improves entity matching accuracy across diverse domains while managing system complexity through a unified architecture rather than multiple specialized systems
Data Source
AI summary
The present application relates to apparatus, systems, and methods for grouping data records based on entities referenced by the data records. The disclosed grouping mechanism can include determining a pair-wise similarity between a large number of data records, and clustering a subset of the data records based on their pair-wise similarity.