Record Linkage Clustering Using Superblocks and K-Mers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing record linkage and entity resolution methods face challenges with long run times, poor accuracy, restricted applicability, and lack of parallel solutions, especially when dealing with diverse and error-prone data sets.
Innovation Solution
The implementation of a superblocking module that uses k-mers and graph-based clustering to efficiently group records, employing blocking algorithms and edit distance thresholds to create accurate clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional record linkage methods are used to handle diverse and error-prone data sets, then accuracy can be maintained through careful comparison, but computation time becomes excessively long
Solution Approach 1:
The patent segments the data processing task into multiple independent blocks, where each block processes a subset of records using distributed computing. This allows parallel processing of record pairs, significantly reducing computation time while maintaining linkage accuracy through systematic comparison of blocking attributes and edit distance calculations
Solution Approach 2:
The patent performs preliminary blocking on records before full comparison, grouping records by blocking attributes (such as name, location, time) and only comparing records within the same block. This preliminary filtering reduces the number of comparisons needed, maintaining accuracy by ensuring relevant records are compared while eliminating unnecessary computations
2Reliability
If comprehensive record comparison is performed to ensure accurate entity resolution, then linkage quality improves, but the system becomes less adaptable to diverse data sources
Solution Approach 1:
The patent implements a universal record linkage framework that can process diverse data sources by configuring blocking attributes and thresholds according to the specific data characteristics. The system maintains linkage quality through configurable comparison parameters while adapting to different data sources through flexible blocking strategies and editable distance metrics
Solution Approach 2:
The patent allows dynamic adjustment of blocking attributes, edit distance thresholds, and matching criteria based on the characteristics of different data sources. This enables the system to maintain high linkage quality across diverse datasets by optimizing parameters for each specific application context
3Productivity
If exact matching algorithms are used for record linkage, then processing speed can be fast for identical records, but accuracy deteriorates when records contain errors or variations
Solution Approach 1:
The patent applies different comparison strategies at different levels: exact matching on blocking attributes (name, location, time) for fast initial filtering, followed by more tolerant edit distance calculations on content attributes. This local differentiation of matching strictness maintains processing speed for clearly matching records while improving accuracy for records with errors or variations
Data Source
AI summary
Methods and apparatuses are described herein for sorting with diverse sets of data. The methods and apparatuses receive a plurality of records from one or more data sources, create superblocks based upon one or more blocking attributes, generate k-mers for all the records based upon a selected k value, perform blocking on the records and place any records with matching k-mers in the appropriate superblocks, define a graph G(V, E) where there is a node per record and connect records via an edge in the graph when the records are found together in at least one of the superblocks and an edit distance between the records is within a given threshold value, and find and output the connected components of graph G(V, E) as final clusters.