Adaptive Dynamic Blocking for Database Record Linkage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing record linkage methods in computer databases face inefficiencies due to poor attribute selection for blocking, leading to excessive computing and memory resource consumption, especially when dealing with large datasets and varying natural groupings of records.

Innovation Solution

An adaptive dynamic blocking technique that determines maximum block sizes and generates candidate pairs based on blocking keys and a duplication factor, dynamically adjusting block sizes to reduce the number of comparisons and optimize resource usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Use of energy by moving object

If blocking is used to reduce the number of comparisons, then computing resources are saved, but poor blocking attribute selection results in a large number of comparisons

Engineering Contradiction:

Improvecomputing resourcesVSAvoidnumber of comparisons

Core Design Contradiction:

Use of energy by moving objectVSProductivity

Solution Approach 1:

The patent implements dynamic blocking by adjusting block sizes and blocking attributes based on the duplication factor calculated from the dataset. The system dynamically determines optimal block sizes and selects blocking attributes iteratively, transforming the static blocking approach into a dynamic one that adapts to data characteristics, thereby reducing the number of comparisons while saving computing resources

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes parameters (block size, blocking attributes) based on the calculated duplication factor. When the duplication factor exceeds a threshold, the system adjusts blocking parameters to optimize the balance between reducing comparisons and maintaining linkage accuracy, directly addressing the contradiction between computing resource usage and comparison efficiency

Inventive Principle:

Principle #35Parameter changes

2Productivity

If blocking is used to reduce the number of comparisons, then processing efficiency is improved, but memory resources are significantly consumed

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidmemory resources

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent segments the dataset into blocks based on blocking attributes and duplication factor. By dividing the large dataset into smaller manageable blocks, the system processes records in segments rather than loading all records into memory simultaneously, thus improving processing efficiency while controlling memory resource consumption

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts block sizes based on the duplication factor to optimize memory usage. When duplication is high, smaller blocks are used to reduce memory footprint; when duplication is low, larger blocks improve processing efficiency. This dynamic segmentation resolves the contradiction between processing efficiency and memory consumption

Inventive Principle:

Principle #15Dynamics

3Ease of manufacture

If a fixed blocking approach is used, then implementation is simple, but it cannot adapt to varying natural groupings of records

Engineering Contradiction:

Improveimplementation simplicityVSAvoidadaptation to record groupings

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent transforms fixed blocking into dynamic blocking by calculating the duplication factor and adjusting blocking attributes and block sizes accordingly. The system iteratively selects blocking attributes and adapts to varying natural groupings in the data, maintaining implementation feasibility while significantly improving adaptability to different dataset characteristics

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system uses feedback from the duplication factor calculation to adjust blocking parameters. The duplication factor serves as a feedback mechanism that informs the system about data characteristics, enabling adaptive blocking that responds to varying natural groupings while maintaining a systematic implementation approach

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20240290445A1Database record linkage using adaptive dynamic blocking

Publication Date: 2024.08.29 OPTUM INC

AI summary

A method includes performing by a host system processor: providing a database including a plurality of records, the database having at least one attribute associated therewith; determining a plurality of maximum block sizes as a plurality of ideal maximum numbers of the plurality of records in a plurality of blocks, respectively, based on a plurality of blocking keys; determining a duplication factor based on a number of unique records of the plurality of records based on all of the at least one attribute; and generating candidate pairs of the plurality of records for linkage based on the plurality of maximum block sizes and the duplication factor.