Trie-Based Match Key Generation for Database Record Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face challenges in achieving optimal recall and performance when matching records, as they need to balance identifying all matching records without degrading performance, which is not feasible with current indexing methods.
Innovation Solution
The database system uses a trie data structure to create optimized match keys for fields with a prefix structure, tokenizing values and creating keys that maximize recall while adhering to performance constraints, by identifying node sequences and associating keys with records, and generating fuzzy variations to account for errors and abbreviations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If narrowly focused match keys are used, then performance is improved, but recall deteriorates
Solution Approach 1:
The patent implements dynamic match key generation by traversing the trie structure and adaptively selecting prefix lengths based on the specific suspect record being processed. Instead of using fixed-length or pre-determined match keys, the system dynamically determines the optimal prefix length for each record by examining the trie node counts and adjusting the match key length accordingly, thereby balancing recall and performance on a per-record basis
Solution Approach 2:
The system changes the parameter of match key length dynamically during the matching process. By traversing the trie structure and examining node counts at different levels, the system adjusts the prefix length parameter to optimize both recall and performance. This allows the match key length to vary based on the data distribution and specific record characteristics rather than being fixed
2Reliability
If every record is treated as a candidate for every suspect, then recall is improved to 100%, but performance deteriorates
Solution Approach 1:
The patent performs preliminary action by pre-building the trie structure from the database records before the matching process. The trie is constructed in advance with tokenized values and node counts, enabling the system to quickly determine optimal match keys during querying without having to process all records exhaustively. This preliminary organization of data allows for efficient matching while maintaining high recall
Solution Approach 2:
The system segments the database records into a hierarchical trie structure during preprocessing, organizing them by tokenized field values with prefix structures. This segmentation enables the matching process to navigate through organized branches rather than comparing records exhaustively, significantly improving performance while maintaining recall through the structured organization
3Reliability
If fuzzy variations are generated, then recall is improved by covering errors and abbreviations, but device complexity increases
Solution Approach 1:
The patent applies local quality by generating fuzzy variations only at specific locations in the trie structure where needed. Instead of uniformly processing all records with complex fuzzy logic, the system selectively applies fuzzy variation generation based on the trie node characteristics and data distribution, thereby improving recall for records with errors or abbreviations while minimizing the overall system complexity
Data Source
AI summary
The system tokenizes values stored by records' fields, creates trie from tokenized values, each branch labeled with tokenized value, each node storing count indicating number of records associated with tokenized value sequence beginning from trie root. The system tokenizes value stored by record field, identifies nodes, beginning from trie root, corresponding to token value sequence associated with tokenized value, until node is identified that stores count that is less than node threshold. The system identifies branch sequence comprising each identified node as record's key, and associates key with node storing count less than node threshold, and record with key. The system tokenizes prospective value stored by prospective record's field, identifies nodes, beginning from trie root, corresponding to another token value sequence associated with tokenized prospective value, until another node is identified that stores another count that is less than node threshold. The system identifies other node's key as prospective record's key, identifies existing record that matches prospective record by using prospective record's key.