String Similarity Search Using Feature Pruning and Guaranteed Rank Threshold
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing string similarity analysis methods require significant computational resources and user tuning, making them inefficient for large datasets and prone to returning irrelevant results, especially when searching for top similarities.
Innovation Solution
A computer-based search tool employing feature pruning, input feature filtering, and a guaranteed rank threshold to reduce the number of similarity metric calculations, using an index with self-information weights to prioritize relevant results and enable search-as-you-type functionality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If similarity metrics are calculated for every string in the collection, then complete search coverage is achieved, but computational resources and time are excessively consumed
Solution Approach 1:
The patent segments the search process into multiple stages: first calculating similarity metrics for a subset of strings to identify candidate results, then calculating metrics for additional strings only if needed to meet the minimum result threshold. This segmentation allows the system to achieve complete search coverage while minimizing unnecessary computations by dividing the workload based on intermediate results.
2Measurement precision
If a minimum similarity metric threshold is set high, then result relevance is improved, but the number of returned results may be insufficient
Solution Approach 1:
The patent implements a dynamic threshold adjustment mechanism where the minimum similarity metric threshold is initially set high to ensure relevance, but the system dynamically lowers the threshold if the number of returned results falls below the minimum threshold. This dynamic adjustment allows the system to maintain high result relevance when possible while ensuring sufficient quantity of results is returned.
3Productivity
If feature pruning and filtering are applied, then computational overhead is reduced, but risk of eliminating relevant features increases
Solution Approach 1:
The patent applies feature pruning and filtering as preliminary actions before full similarity metric calculation. By pre-processing and eliminating obviously irrelevant features based on basic string comparison and feature analysis, the system reduces computational overhead while maintaining result accuracy. The pruning is performed in advance of detailed similarity calculations, allowing efficient elimination of non-promising candidates without sacrificing the ability to identify truly relevant results.
Data Source
AI summary
A string analysis tool for calculating a similarity metric between an input string and a plurality of strings in a collection to be searched. The string analysis tool may include optimizations that may reduce the number of calculations to be carried out when calculating the similarity metric for large volumes of data. In this regard, the string analysis tool may represent strings as features. As such, analysis may be performed relative to features (e.g., of either the input string or plurality of strings to be searched) such that features from the strings may be eliminated from consideration when identifying candidate strings from the collection for which a similarity metric is to be calculated. The elimination of features may be based on a minimum similarity metric threshold, wherein features that are incapable of contributing to a similarity metric above the minimum similarity metric threshold are eliminated from consideration.