Database Record Linkage Tiling for Memory Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database record linkage methods are inefficient due to the large number of database readings required and varying block sizes, leading to memory resource wastage and out-of-memory errors when comparing all possible pairs of records.
Innovation Solution
The system and method optimize database access by segmenting data into smaller segments, pairing them into quadrants, and detecting duplicates within each quadrant, ensuring that each pair of records is available in memory and minimizing database readings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all possible pairs of records are compared to find duplicates, then complete record linkage is achieved, but the number of database readings becomes excessively large and processing time increases
Solution Approach 1:
The patent divides the database into multiple blocks and segments the comparison task into comparing records within the same block first, then comparing across blocks only when necessary. This segmentation reduces the total number of comparisons from O(n²) to a much smaller number by leveraging the blocking structure, thereby reducing processing time while maintaining complete record linkage through the two-pass approach.
2Productivity
If blocking keys are used to load records into memory for comparison, then memory access is optimized, but block sizes vary widely causing memory resource waste or out-of-memory errors
Solution Approach 1:
The patent dynamically adjusts the number of blocks and records per block based on available memory resources. The system calculates optimal block parameters before execution, allowing the blocking structure to adapt to different memory capacities. This dynamic configuration ensures efficient memory utilization without wasting resources on overly small blocks or causing out-of-memory errors with excessively large blocks.
3Quantity of substance
If the database is divided into many small blocks for comparison, then memory usage is reduced, but the number of required database readings increases significantly
Solution Approach 1:
The patent performs preliminary blocking and sorting of records before the actual duplicate detection process. By pre-organizing records into blocks with the same blocking key, the system ensures that records requiring comparison are already positioned efficiently in memory. This preliminary action reduces the need for repeated database readings during the comparison phase, as most comparisons can be performed using already-loaded block data.
Data Source
AI summary
A system and method for optimizing database access for record linkage by tiling the space of record pairs are provided, the system including a processor, a segmentation and pairing unit in signal communication with the processor for segmenting database data into data segments and pairing the data segments into data quadrants, and a duplicate detection unit in signal communication with the processor for detecting duplicates for each quadrant; and the method including receiving database data, segmenting the database data into data segments, pairing the data segments into data quadrants, and detecting duplicates for each quadrant.