Fuzzy Token Generation for Obscure Text Matching Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing fuzzy text matching techniques perform inconsistently across different types of texts and linguistic domains, often providing poor results in certain scenarios, leaving users without effective text matching solutions.
Innovation Solution
A computer-implemented method and system that generates fuzzy tokens from glossary terms, calculates similarity scores, and matches input terms with glossary terms using n-grams and fuzzy tokens, to identify candidate matches and quantify their similarity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional fuzzy text matching techniques are used, then the matching process is simple, but the matching accuracy deteriorates on obscure texts and certain linguistic domains
Solution Approach 1:
The patent segments the text matching process into multiple stages: generating n-grams from input text, creating fuzzy tokens from glossary terms, and systematically comparing these segments. This segmentation allows the system to handle obscure texts by breaking them down into manageable units that can be matched against glossary terms even when the overall text structure is unrecognized.
Solution Approach 2:
The patent introduces an intermediary glossary component that acts as a mediator between the input text and the matching algorithm. The glossary provides domain-specific terms and their variations, serving as a reference layer that improves matching accuracy for obscure texts by providing expected patterns and alternatives that the fuzzy token generator can match against.
2Adaptability or versatility
If fuzzy token generation with multiple variations is implemented, then the coverage of obscure texts improves, but the computational complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-generating fuzzy tokens from glossary terms before the actual matching process. This includes creating multiple variations of each glossary term (abbreviations, misspellings, partial matches) in advance and storing them in a searchable structure, which speeds up the subsequent matching process despite the increased number of tokens.
Solution Approach 2:
The system dynamically adjusts the fuzzy token generation based on the input text characteristics. The tokenizer adapts to different text formats and the fuzzy token generator creates variations appropriate for the specific linguistic domain being processed, allowing the system to handle diverse text types efficiently without generating all possible variations for every case.
3Measurement precision
If multiple scoring metrics are calculated, then the match quality assessment improves, but the processing time increases
Solution Approach 1:
The patent implements self-service through automated score aggregation and threshold-based filtering. The system automatically combines multiple scoring metrics (n-gram overlap, fuzzy token similarity, glossary term frequency) into a composite match score, and uses predefined thresholds to quickly filter out low-quality matches without requiring manual review or extensive post-processing.
Data Source
AI summary
A method comprising: obtaining multiple glossary terms each comprising one or more words; generating multiple fuzzy tokens from each word of each of the glossary terms; calculating a similarity score for each of the fuzzy tokens, the similarity score denoting a similarity between the respective fuzzy token and its respective word; obtaining multiple input terms to be matched with the multiple glossary terms; separating each of the input terms into multiple input tokens; generating multiple n-grams from each of the input tokens; comparing the n-grams with the fuzzy tokens, to output a list of matching n-grams and fuzzy tokens; based on the list of matching n-grams and fuzzy tokens, identifying, from the glossary terms, candidate glossary term matches for each of the input terms; and calculating one or more scores that quantify the match between each of the candidate glossary term matches and its respective input term.