Fuzzy String Matching Using Prefix Tree Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing fuzzy searching algorithms, such as those using Levenshtein distance and Levenshtein automaton, are computationally intensive and resource-heavy, leading to long processing times and high energy consumption when dealing with large data sets.
Innovation Solution
A method utilizing prefix tree data structures and a depth-first search algorithm to identify matches and approximate matches between two sets of strings, allowing for user-definable maximum error and efficient processing by traversing trees while skipping non-matching nodes within a defined distance, thereby reducing the number of processing operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pairwise matching using Levenshtein distance is used to perform fuzzy searching, then matching accuracy is improved, but processing time increases significantly
Solution Approach 1:
The patent segments the search space by organizing strings into prefix trees (tries), where common prefixes are shared across multiple strings. This segmentation allows the algorithm to avoid redundant comparisons of identical prefix portions, significantly reducing the number of pairwise comparisons needed while maintaining Levenshtein distance accuracy for the remaining unique portions of strings.
Solution Approach 2:
The patent performs preliminary actions by pre-processing the input strings into prefix tree data structures before executing the fuzzy matching algorithm. This pre-organization of data into hierarchical prefix groups enables the subsequent matching process to skip over large numbers of obviously non-matching string pairs, reducing processing time while preserving matching accuracy.
2Reliability
If comprehensive pairwise comparisons are performed to ensure all matches are found, then matching completeness is improved, but computational resource usage increases
Solution Approach 1:
By segmenting the comparison task into prefix-based groups, the algorithm processes only relevant string pairs within each prefix group rather than performing exhaustive pairwise comparisons across all strings. This segmentation maintains matching completeness within the defined distance threshold while dramatically reducing the total number of comparisons and associated computational resources.
Solution Approach 2:
The patent applies partial action by implementing a maximum distance threshold k that limits the depth of exploration in the prefix trees. This allows the algorithm to find all matches within the acceptable error margin without performing excessive comparisons beyond the threshold, optimizing the balance between completeness and resource consumption.
3Productivity
If exact matching algorithms are used, then processing speed is improved, but flexibility to handle approximate matches deteriorates
Solution Approach 1:
The patent introduces dynamics by making the matching criterion adjustable through the maximum distance parameter k. The algorithm can dynamically adapt between exact matching (k=0) and approximate matching (k>0) based on user requirements, while the prefix tree structure maintains efficient processing speeds for both modes by organizing data to minimize comparisons regardless of the threshold setting.
Data Source
AI summary
A method, system and computer program product is disclosed for fuzzy searching. The method, which may be performed by one or more processors, may comprise providing a first prefix tree data structure representing a first data set comprising a first plurality of strings, and providing a second prefix tree data structure representing a second data set comprising a second plurality of strings. The first and second prefix tree data structures may each comprise nodes representing each character and edges connecting prefix nodes to one or more suffix nodes to represent each subsequent character in the string. A search may be performed to identify all matches between the first and second plurality of strings and also approximate matches between the first and second plurality of strings within a maximum distance k, wherein the search comprises traversing the first prefix tree data structure using a depth-first search algorithm to identify matches and approximate matches in the second prefix tree data structure.