Parallel Record Linkage via Smart Indexing and Hierarchical Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current record linkage methods are inefficient for large datasets due to sequential processing and lack of tolerance for node failures in distributed networks, making it difficult to handle big data effectively.
Innovation Solution
A parallelized record linkage method that uses indexing and hierarchical clustering, allowing for distributed clustering and aggregation of records across multiple nodes, with a mechanism for sub-cluster matching and tolerance for node failures, ensuring the process can continue even if some nodes fail.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential record linkage processing is used, then the system is simple to implement, but the processing efficiency is low for large datasets
Solution Approach 1:
The patent divides the record linkage task into multiple independent clustering operations that can be executed in parallel across different nodes. Each node processes a subset of records independently, creating clusters that are later merged. This segmentation enables parallel processing while maintaining manageable complexity at each node level.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by distributing the clustering task across multiple computational nodes simultaneously. Instead of sequential single-node processing, the system operates across multiple dimensions of computation, achieving speedup proportional to the number of nodes while using standardized clustering algorithms at each node.
2Productivity
If distributed parallel processing is implemented, then processing speed improves, but the system becomes vulnerable to node failures
Solution Approach 1:
The patent implements fault tolerance by having nodes exchange cluster information and perform cross-validation. Each node receives cluster data from other nodes and verifies consistency, allowing the system to detect and compensate for failed nodes before they cause complete processing failure. This beforehand cushioning ensures continued operation despite individual node failures.
3Productivity
If records are clustered across multiple nodes, then processing capacity increases, but coordinating and aggregating results becomes complex
Solution Approach 1:
The patent simplifies coordination by having all nodes exchange their cluster results and merge them into a unified set of clusters. Rather than complex distributed algorithms for maintaining consistency, the system uses straightforward cluster exchange and merging operations. Duplicate clusters are identified and consolidated, producing a final unified clustering result with minimal coordination overhead.
Data Source
AI summary
Embodiments include a computer-implemented method including identifying, by a primary computer device, a plurality of records, each record having one or more attributes; standardizing, by the primary computer device, each of the plurality of records; assigning, by the primary computer device, an index to one or more of the one or more attributes; providing, by the primary computer device, instructions for clustering the standardized plurality of records in parallel into one or more clusters, each cluster including records having the same index, the one or more clusters being in a group; receiving, by the primary computer device, one or more groups, each group including one or more clusters sharing a same index; and linking one or more of the plurality of records in a cluster with another one or more of the plurality of records in another cluster within a same group.