Intersection-Based Dynamic Blocking for Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods face challenges in efficiently identifying and eliminating redundant information from large datasets with heterogeneous data sources, as comparing all possible pairs of records to detect duplicates becomes intractable due to the complexity and size of the data sets.
Innovation Solution
The implementation of intersection-based dynamic blocking, which iteratively reduces block sizes by identifying non-empty intersections of oversized input blocks, allowing for detailed similarity analysis and redundancy elimination through pivot operations and transformation functions, enabling efficient generation of block identifiers and elimination of duplicate combinations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all possible pairs of records are compared to identify duplicates, then detection precision is improved, but computing time and complexity become intractable
Solution Approach 1:
The patent divides the large dataset into multiple blocks, where each block contains a subset of records. Instead of comparing all possible pairs across the entire dataset, the system performs similarity analysis only within each block, significantly reducing the number of comparisons required while maintaining duplicate detection capability.
Solution Approach 2:
The patent implements a two-stage process: first performing blocking to group similar records into blocks before conducting detailed similarity analysis. This preliminary grouping action reduces the search space for duplicate detection, allowing the system to achieve high precision without intractable computing time.
2Productivity
If block sizes are reduced to enable detailed similarity analysis, then processing efficiency is improved, but memory requirements increase
Solution Approach 1:
The patent segments the dataset into blocks of manageable size, allowing detailed similarity analysis to be performed on each block independently. This segmentation enables processing efficiency to improve while memory requirements remain controlled, as each block fits within available memory constraints.
Solution Approach 2:
The patent introduces a new dimension of organization by creating blocks with hierarchical identifiers. This block-based structure adds a layer of abstraction that allows efficient memory management, enabling the system to process large datasets by working with smaller block units that can be loaded into memory sequentially.
3Productivity
If intersection operations are performed on oversized blocks, then block size reduction is achieved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary blocking to create initial groups of similar records before conducting intersection operations. This preliminary organization reduces the complexity of subsequent intersection computations, as the data is already partially structured and filtered, allowing block size reduction to proceed efficiently.
Solution Approach 2:
The patent uses block identifiers and intermediate blocking structures as mediators to simplify intersection operations. Instead of directly computing intersections on raw oversized blocks, the system uses intermediate block representations that reduce computational complexity while achieving the desired block size reduction.
Data Source
AI summary
Block size reduction iterations are performed on a plurality of blocks of records until a block size criterion is met. An iteration comprises identifying, from a first collection of blocks, using one or more pivot operations, a set of combinations of oversized blocks such that at least one record belongs to all blocks of a combination. A new block comprising records that are members of each block of a first combination of the set is included in a second collection of blocks to be examined in a subsequent iteration. On at least one block created in an iteration, analysis operations are performed.