Database N-gram Indexing for Approximate String Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database technologies are limited in identifying near-duplicate merchant names and locations due to variations in data entry formats, leading to inefficiencies in matching and updating merchant records within payment card systems, and existing approximate string matching solutions are often costly, domain-specific, or external to the database system.
Innovation Solution
A method for approximate string matching within a database system that involves generating n-gram representations for candidate and reference character strings, determining similarities, and indexing the candidate string based on these similarities, allowing for efficient retrieval of matches without computing similarity metrics for the entire database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional exact string matching is used in database systems, then implementation simplicity is maintained, but the ability to identify near-duplicate merchant records variesing in data entry formats is severely limited
Solution Approach 1:
The patent segments character strings into n-gram units (substrings of length n). By breaking down merchant names and addresses into overlapping character sequences, the system can compare individual n-grams rather than entire strings, enabling detection of partial matches and near-duplicates while maintaining computational efficiency through indexed n-gram storage.
Solution Approach 2:
The patent introduces n-gram representations as an intermediary between the original character strings and the matching process. Instead of directly comparing full merchant records, the system converts strings to n-gram sets and computes similarity based on n-gram overlap, serving as a mediator that transforms the matching problem into a more manageable form.
2Measurement precision
If approximate string matching algorithms are implemented to identify near-duplicate merchant records, then matching accuracy for varied data formats is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing n-gram representations of merchant records in the database before actual matching operations. This advance preparation allows the system to quickly retrieve and compare n-gram sets during matching without performing complex string operations in real-time, significantly improving processing throughput.
Solution Approach 2:
The patent implements self-service through automated n-gram generation and similarity computation. The system automatically converts merchant records to n-gram representations and computes similarity scores without requiring manual intervention, enabling high-volume processing while maintaining consistent matching quality across all records.
3Reliability
If comprehensive string matching is performed on all merchant records in the data warehouse, then complete coverage of potential duplicates is achieved, but computational overhead becomes prohibitive for large databases
Solution Approach 1:
The patent extracts only the essential matching information by converting full merchant records to compact n-gram representations. This extraction process removes redundant data while preserving the key characteristics needed for duplicate detection, reducing the amount of data that must be processed and stored while maintaining matching completeness.
Solution Approach 2:
The patent changes the parameter representation from full character strings to n-gram frequency vectors. This parameter transformation enables efficient computation of similarity metrics (such as Jaccard similarity) by operating on numeric vectors rather than text, significantly reducing computational overhead while maintaining the ability to identify all potential duplicates.
4Measurement precision
If third-party approximate string matching solutions are used, then matching capability is enhanced, but cost increases and domain-specific customization requirements arise
Solution Approach 1:
The patent implements a universal n-gram matching framework that can handle various types of string comparison needs within a single system. The same n-gram infrastructure serves multiple purposes including duplicate detection, similarity scoring, and record linkage across different data formats and domains, eliminating the need for separate specialized solutions and reducing overall system cost.
Data Source
AI summary
A computer-based method for character string matching of a candidate character string with a plurality of character string records stored in a database is described. The method includes a) identifying a set of reference character strings in the database, the reference character strings identified utilizing an optimization search for a set of dissimilar character strings, b) generating an n-gram representation for one of the reference character strings in the set of reference character strings, c) generating an n-gram representation for the candidate character string, d) determining a similarity between the n-gram representations, e) repeating steps b) and d) for the remaining reference character strings in the set of identified reference character strings, and f) indexing the candidate character string within the database based on the determined similarities between the n-gram representation of the candidate character string and the reference character strings in the identified set.