Geo-Gridded Entity Resolution for Faster Database Reconciliation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Merging or reconciling datasets with overlapping information is a slow and resource-intensive process due to the inefficiency in sorting through large quantities of data records to identify and remove duplicates.
Innovation Solution
A computer-based method that utilizes geo-gridding to map geographical areas based on latitude and longitude data, determining candidate and actual matching records through trigram similarity thresholds, thereby efficiently identifying and associating matching data records across multiple datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exhaustive record comparisons are performed to identify duplicates, then matching precision is improved, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent segments the geographical space into a grid of discrete cells based on latitude and longitude ranges. Each record is assigned to specific grid cells, and only records within the same or adjacent cells are compared. This spatial segmentation dramatically reduces the number of comparisons needed while maintaining matching precision, as records from different geographical regions are automatically filtered out.
Solution Approach 2:
The patent performs preliminary action by pre-calculating and storing grid cell assignments for all records before the actual matching process. The geographical space is divided into grid cells in advance, and records are pre-mapped to their corresponding cells. This preliminary organization enables efficient filtering during matching, avoiding the need for exhaustive comparisons while ensuring accurate identification of potential duplicates.
2Reliability
If all data records are sorted through to remove duplicates, then data quality is improved, but computational resource consumption increases
Solution Approach 1:
The patent divides the entire dataset into smaller subsets based on geographical grid cells. Instead of comparing all records against each other, the system only compares records within the same or adjacent grid cells. This segmentation approach maintains data quality by ensuring thorough local comparison while dramatically reducing overall computational resource consumption.
Solution Approach 2:
The patent introduces grid cells as an intermediary structure between the full dataset and the comparison process. Grid cells act as mediators that organize and filter records, allowing the system to focus computational resources only on records that are geographically proximate. This intermediary structure enables high-quality duplicate removal without exhausting computational resources.
3Quantity of substance
If traditional duplicate removal methods are used, then completeness of data merging is improved, but processing efficiency deteriorates
Solution Approach 1:
The patent segments the merging process into two stages: first, geographical filtering using grid cells to identify potential matches, and second, detailed comparison only for records within the same or adjacent grid cells. This segmentation maintains merging completeness by ensuring all geographically relevant records are considered, while significantly improving processing efficiency by avoiding unnecessary comparisons.
Solution Approach 2:
The patent performs preliminary geographical filtering by assigning records to grid cells before conducting detailed duplicate comparisons. This preliminary action organizes the data in a way that enables efficient processing, as the system only needs to compare records that have been pre-identified as potentially matching based on their grid cell assignments.
Data Source
AI summary
To facilitate efficient entity resolution, systems and methods include a first dataset is received from a first database associated with a first entity and a second dataset is received from a second database associated with a second entity. A geo-grid is mapped to a geographical area covering the first entity data records and the second entity data records. A grid matching area in the geo-grid is generated for each first entity data record based on latitude data and longitude data of each first entity data record. Candidate matching records are determined from the second entity data records based on respective grid matching areas. Actual matching records are determined from the candidate matching records based on a threshold for a trigram similarity between each candidate matching record and the respective first entity data record. The actual matching records are associated with the respective first entity data record in the first database.