Record Linkage via Elimination of Highly Overlapped Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data lakes face inefficiencies in managing and processing large volumes of overlapping records, leading to system failures due to the creation of too many blocks during dynamic blocking, which hinders effective record linkage and deduplication.
Innovation Solution
Implementing techniques such as intersection dynamic blocking, block elimination, and demotion to identify and eliminate highly overlapping blocks, using methods like Jaccard similarity and subsumption analysis to reduce block sizes and prevent system overload, while automatically configuring blocking parameters based on data statistics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dynamic blocking is used to identify duplicate records in a data lake, then record linkage capability is improved, but the number of blocks created becomes excessively large leading to system failure
Solution Approach 1:
The patent applies segmentation by dividing the data processing task into multiple passes. Instead of creating all possible blocks at once, the system performs iterative passes where each pass processes a subset of blocks. This segmentations the overwhelming task of handling all blocks simultaneously into manageable chunks, preventing system failure while maintaining record linkage capability.
Solution Approach 2:
The patent implements preliminary action by performing a first pass of dynamic blocking to identify and eliminate highly overlapping blocks before proceeding to subsequent processing passes. This preliminary elimination reduces the total number of blocks that need to be processed in later passes, preventing the system from being overwhelmed while preserving the ability to find duplicate records.
2Measurement precision
If all blocks are processed to ensure complete record matching, then matching accuracy is improved, but processing time and system resources are excessively consumed
Solution Approach 1:
The patent applies partial action by processing blocks in multiple passes rather than attempting to process all blocks simultaneously in a single pass. Each pass handles a portion of the blocks, eliminating highly overlapping blocks in earlier passes and reducing the workload for subsequent passes. This approach maintains matching accuracy by ensuring thorough processing while significantly reducing total processing time compared to handling all blocks at once.
3Productivity
If blocking parameters are set to capture all potential matches, then recall rate is improved, but block sizes become unmanageably large causing system overload
Solution Approach 1:
The patent implements dynamics by using adaptive blocking parameters that adjust across multiple passes. In the first pass, parameters are set to be more restrictive, creating smaller initial blocks. As highly overlapping blocks are eliminated, subsequent passes can use less restrictive parameters to capture additional matches without creating unmanageably large blocks. This dynamic adjustment maintains high recall rates while preventing system overload.
Solution Approach 2:
The patent applies preliminary action by performing an initial pass with conservative blocking parameters to eliminate the most highly overlapping blocks before subsequent passes use more aggressive parameters. This preliminary reduction in block overlap prevents the creation of unmanageably large blocks in later passes while still capturing the majority of duplicate records, balancing recall rate with system manageability.
Data Source
AI summary
Techniques for scaling record linkage via elimination of highly overlapped blocks are described. A method for scaling record linkage via elimination of highly overlapped blocks includes identifying a first plurality of blocks based at least on a plurality of records stored in a storage service of a provider network, identifying a plurality of sets of matching blocks from the first plurality of blocks, deleting the plurality of sets of matching blocks except for a first block from each set from the plurality of sets of matching blocks, and iteratively performing dynamic blocking based at least on the first block to generate subsequent pluralities of blocks until the subsequent pluralities of blocks are below a threshold size.