Parallel Elimination of Subsumed Blocks in Data Lakes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data lakes face inefficiencies in managing highly overlapping records, leading to system failures due to the creation of too many blocks during dynamic blocking, which overwhelms processing capabilities.
Innovation Solution
Implementing techniques such as intersection dynamic blocking, block elimination, and demotion to identify and eliminate highly overlapping blocks, using methods like Jaccard similarity and subsumption analysis to reduce block sizes and prevent system overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dynamic blocking is used to identify duplicate records in large databases, then record linkage capability is improved, but the number of blocks created becomes excessively large leading to system failure
Solution Approach 1:
The patent segments the blocking process into multiple hierarchical levels (first-level blocking, second-level blocking, third-level blocking). Each level processes blocks of decreasing size, with the system dividing the original large dataset into manageable segments that are processed sequentially. This multi-level segmentation prevents the creation of excessively large block sets while maintaining comprehensive duplicate record identification.
Solution Approach 2:
The patent extracts and eliminates subsumed blocks from the processing pipeline. By identifying blocks that are subsets of other blocks (using subsumption analysis) and removing them from further processing, the system reduces the total number of blocks that need to be handled while preserving all necessary duplicate record relationships.
2Measurement precision
If intersection dynamic blocking is applied to find duplicate records, then matching accuracy is improved, but processing time increases due to excessive block creation
Solution Approach 1:
The patent performs preliminary blocking operations at multiple levels before final duplicate record identification. By pre-processing the data through first-level, second-level, and third-level blocking, the system prepares the data in an optimized structure that enables accurate matching while reducing the computational burden during the actual duplicate identification phase.
Solution Approach 2:
The patent extracts and removes subsumed blocks that would otherwise be processed unnecessarily. By identifying and eliminating blocks that are subsets of other blocks using subsumption analysis, the system removes redundant processing steps while preserving all necessary matching relationships, thereby reducing total processing time.
3Reliability
If multiple blocking levels are implemented to reduce block size, then system stability is improved, but device complexity increases
Solution Approach 1:
The patent divides the blocking process into three distinct hierarchical levels, each with specific block size thresholds and processing rules. This segmentation provides system stability by ensuring that no single blocking operation handles excessively large datasets, while the modular structure of multiple independent levels actually simplifies the overall system design compared to attempting to handle all data in a single blocking operation.
Data Source
AI summary
Techniques for Scalable parallel elimination of approximately subsumed sets are described. A method for Scalable parallel elimination of approximately subsumed sets includes identifying a first plurality of blocks based at least on a plurality of records stored in a storage service of a provider network, determining a plurality of subsumption relationships between blocks from the first plurality of blocks, retaining a first subset of the first plurality of blocks and demoting a second subset of the first plurality of blocks based at least on the plurality of subsumption relationships, and iteratively performing dynamic blocking based at least on the first subset of the plurality of matching blocks and the second subset of the plurality of matching blocks to generate a subsequent pluralities of blocks.