Fuzzy Match Graph for Database Record Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing record matching and deduplication techniques in large datasets are inefficient, leading to prohibitive time and hardware resource requirements due to their inability to scale with the number of records and fields, resulting in duplicate data entries that waste resources and confuse users.
Innovation Solution
A heterogeneous graph structure is used to link records with common match key values derived from fuzzy field matching, reducing computational load by constructing a fuzzy match graph and assigning unique identifiers to subgraphs, allowing for efficient identification and reconciliation of related records without requiring extensive database queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional record matching and deduplication techniques are used to identify and reconcile related records, then matching accuracy is improved, but computational time and hardware resource requirements increase prohibitively as the number of records and fields increases
Solution Approach 1:
The patent segments the computational task by dividing records into blocks and processing them in pairs, where each block contains a subset of records. This segmentation allows the system to handle large datasets by breaking them into manageable units, computing match keys for each block separately, and combining results incrementally, thereby reducing the overall computational time and resource requirements while maintaining matching accuracy.
2Measurement precision
If traditional record matching and deduplication techniques are used to identify and reconcile related records, then matching accuracy is improved, but hardware computing resource requirements increase prohibitively as the number of records and fields increases
Solution Approach 1:
The patent segments the computational task by dividing records into blocks and processing them in pairs, where each block contains a subset of records. This segmentation allows the system to handle large datasets by breaking them into manageable units, computing match keys for each block separately, and combining results incrementally, thereby reducing the overall computational time and resource requirements while maintaining matching accuracy.
3Quantity of substance
If the number of data sources and amount of data increases to provide comprehensive information, then data completeness is improved, but the likelihood of duplicate or redundant data entries increases
Solution Approach 1:
The patent applies preliminary action by computing match keys for each record block before performing the actual matching operation. The match keys are computed in advance and stored, allowing the system to quickly compare and identify duplicate records during the matching phase. This preliminary computation of match keys enables efficient detection and elimination of duplicate entries while maintaining data completeness from multiple sources.
Data Source
AI summary
Devices, systems and methods are provided for managing records. One method involves constructing a fuzzy match graph data structure with edges between different field values for a fuzzy matched field of a plurality of records in accordance with a matching rule, assigning unique identifiers to different subgraphs of field values within the fuzzy match graph data structure, determining a match key value associated with a respective record based on a respective unique identifier assigned to a respective subgraph of field values including a respective field value for the fuzzy matched field of the respective record, constructing a heterogenous graph data structure having edges between the respective match key values associated with the respective records of the plurality of records, identifying a subset of related records using the heterogenous graph data structure, and performing one or more actions on the subset of related records at a database system.