Surrogate Blocking Keys for Record Linkage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing record linkage technologies face challenges in efficiently linking large amounts of unstructured data to structured records in real-time, particularly due to high computational complexity and resource requirements when dealing with hundreds of millions of entries.
Innovation Solution
A computer-implemented method using locality sensitive hashing to assign initial and final surrogate identifier values, ensuring uniform distribution within blocks, allowing for efficient record linkage by reducing the need to compare incoming records with every record in the reference data set, and enabling lookup in constant time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If record linkage is performed by comparing incoming records with every record in the reference data set, then matching accuracy is improved, but computational complexity and processing time increase prohibitively
Solution Approach 1:
The reference data set is divided into multiple blocks based on hashing of blocking keys. Each block contains a subset of records that are potential matches for incoming records. This segmentation reduces the number of comparisons needed from O(N) to O(N/B) where B is the number of blocks, significantly lowering computational complexity while maintaining matching accuracy through targeted comparisons within relevant blocks.
Solution Approach 2:
Blocking keys serve as intermediary elements that indirectly link incoming records to potential matches in the reference data set. Instead of directly comparing all records, the system uses blocking keys (derived from hashing blocking attributes) as mediators to identify candidate blocks, reducing the search space while preserving the ability to find accurate matches.
2Measurement precision
If blocking functions are applied to multiple fields to obtain multiple blocking dimensions, then candidate record filtering is improved, but computational cost increases to the order of the sum of cardinality of each block
Solution Approach 1:
The system segments the blocking process into hierarchical levels: first applying blocking on primary keys to create coarse blocks, then using secondary blocking keys to further segment within those blocks. This multi-level segmentation allows the system to apply multiple blocking dimensions without computing the full intersection of all blocks, maintaining processing speed while improving candidate filtering accuracy.
Solution Approach 2:
The system performs preliminary blocking operations during data loading and maintenance phases, pre-computing and storing block assignments for all reference records. This preliminary action ensures that during real-time record linkage operations, the system only needs to perform simple block lookups rather than re-computing complex intersections, maintaining high processing speed even with multiple blocking dimensions.
3Productivity
If the reference data set is divided into blocks, then computational complexity is reduced, but uniform distribution of identifiers within blocks must be ensured for optimal performance
Solution Approach 1:
The system creates surrogate identifier copies that map original record identifiers to block-local identifiers. These surrogate identifiers are uniformly distributed within each block and serve as simplified references during record linkage operations. This copying approach maintains processing efficiency while simplifying identifier management, as the uniform distribution property is built into the surrogate identifier generation process rather than requiring complex post-processing.
Data Source
AI summary
A computer-implemented method and a related system for record linkage of an incoming record to a reference data set may be provided. The method comprises providing a reference data set comprising a plurality of records, each record comprising a plurality of attributes. The method comprises further assigning each of the plurality of records an initial surrogate identifier value, assigning a plurality of block identifiers to each of the records by applying a locality sensitive hashing function to a predefined attribute of the records, resulting in the plurality of the block identifiers, and determining a final surrogate identifier value to each of the records assigned to one of the blocks such that the final surrogate identifier values in each block are uniformly distributed.