Approximate String Matching via Clustering and N-Gram Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database technologies are limited in their ability to efficiently perform approximate string matching for merchant names and locations, leading to duplicate entries and inefficiencies in data management, particularly in the payment card industry, where variations in data formats cause challenges in identifying similar records.
Innovation Solution
A method involving clustering, n-gram representation, and genetic algorithms is used to generate reference strings and index candidate strings within a database, allowing for efficient approximate matching by determining similarities without computing metrics across the entire database, utilizing techniques like clustering, spanning trees, and genetic algorithms to identify dissimilar reference strings and create binary indices for fast matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current database technology is used for string matching, then the system is simple to operate, but the matching accuracy is insufficient leading to duplicate entries
Solution Approach 1:
The patent segments the string matching process into multiple stages: clustering similar strings together, selecting representative reference strings from each cluster, and then comparing candidate strings against these references. This segmentation transforms a single complex comparison task into manageable stages, improving matching accuracy while controlling system complexity through structured processing.
Solution Approach 2:
The patent performs preliminary clustering and reference string selection before actual matching operations. By pre-processing the data to identify and group similar strings, then selecting representative references in advance, the system prepares the matching environment to achieve higher accuracy without requiring complex real-time computations during the matching phase.
2Measurement precision
If exhaustive similarity computations are performed across the entire database, then matching accuracy is maximized, but processing time increases significantly
Solution Approach 1:
The patent extracts a small subset of reference strings from the full database that represent the diversity of string variations. Instead of comparing candidate strings against all five million database entries, the system extracts and uses only the pre-selected reference strings from each cluster, dramatically reducing computation time while preserving matching accuracy through representative sampling.
Solution Approach 2:
The patent applies partial action by performing similarity computations only against a selected subset of reference strings rather than the entire database. This partial comparison approach is sufficient to achieve accurate matching results without the excessive computational burden of exhaustive comparisons, balancing accuracy requirements with processing efficiency.
3Measurement precision
If third party approximate string matching solutions are used, then matching capability is improved, but the solution is external to the database system and costly
Solution Approach 1:
The patent implements a universal matching framework that can handle various string comparison needs within the database system itself. The clustering and reference string approach creates a multi-functional solution that serves approximate matching, duplicate detection, and data quality improvement needs, replacing the need for external specialized solutions while maintaining flexibility and adaptability.
Solution Approach 2:
The patent introduces reference strings as intermediary elements between candidate strings and the full database. These reference strings act as mediators that capture the essential characteristics of string variations, enabling accurate matching through indirect comparison rather than direct exhaustive searching, thus improving capability while maintaining system simplicity.
Data Source
AI summary
A computer-based method for character string matching of a candidate character string with a plurality of character string records stored in a database is described. The method includes performing a clustering operation on at least a portion of the plurality of character string records, the clustering operation generating a plurality of clusters, each cluster comprising a plurality of character strings from the plurality of character string records, the plurality of character strings in each cluster are determined to be similar with respect to each other based on at least one characteristic of the plurality of character strings. The method also includes generating a set of reference character strings that are selected from the plurality of character strings in each cluster, generating an n-gram representation for one of the reference character strings in the set of reference character strings, and generating an n-gram representation for the candidate character string.