Variable Length Gram Indexing for Approximate String Query Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing string matching algorithms using fixed-length grams face performance issues due to large index sizes, which hinder real-time performance and memory efficiency, especially in applications like web search and database systems, as they require substantial memory to handle numerous queries efficiently.
Innovation Solution
The VGRAM technique employs variable-length grams judiciously chosen based on frequency analysis to reduce index size and improve query performance, allowing for efficient approximate string matching without significant modifications to existing algorithms, by generating high-quality grams and determining their relationship with edit distance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fixed-length grams are used to build indexing structures, then string matching can be performed, but the index size becomes very large causing poor real-time performance
Solution Approach 1:
The patent changes the parameter of gram length from fixed to variable, selecting grams of different lengths based on their frequency and quality. This allows the indexing structure to use fewer, more informative grams, significantly reducing index size while maintaining string matching capability and improving query response time.
Solution Approach 2:
The patent extracts only the most useful grams (high-quality grams with lower frequencies) from the complete set of possible grams. By taking out only the essential grams needed for effective string matching, the index size is reduced while preserving the ability to perform accurate string matching queries.
2Reliability
If fixed-length grams are used for indexing, then approximate string matching is enabled, but memory consumption increases substantially
Solution Approach 1:
The patent changes the parameter of gram length from fixed to variable, dynamically selecting gram lengths based on frequency analysis. This parameter change enables the system to use fewer grams in the index, substantially reducing memory consumption while maintaining approximate string matching capability.
Solution Approach 2:
The patent extracts only the most necessary grams for indexing by filtering out low-quality grams with high frequencies. This extraction approach reduces the quantity of data stored in memory while preserving the essential functionality for approximate string matching.
3Measurement precision
If more grams are included in the index to improve matching accuracy, then query precision increases, but index size and processing time increase
Solution Approach 1:
The patent changes the parameter of gram selection criteria from using all grams to using only high-quality grams with specific frequency characteristics. This selective approach maintains measurement precision for string similarity while reducing the number of grams processed, thereby decreasing query processing time.
Solution Approach 2:
The patent extracts only the most informative grams for indexing, removing redundant and less useful grams. This extraction maintains the accuracy of string similarity measurement by keeping only the essential grams, while reducing processing time by eliminating unnecessary grams from the index.
Data Source
AI summary
A computer process, called VGRAM, improves the performance of these string search algorithms in computers by using a carefully chosen dictionary of variable-length grams based on their frequencies in the string collection. A dynamic programming algorithm for computing a tight lower bound on the number of common grams shared by two similar strings in order to improve query performance is disclosed. A method for automatically computing a dictionary of high-quality grams for a workload of queries. Improvement on query performance is achieved by these techniques by a cost-based quantitative approach to deciding good grams for approximate string queries. An approach for answering approximate queries efficiently based on discarding gram lists, and another is based on combining correlated lists. An indexing structure is reduced to a given amount of space, while retaining efficient query processing by using algorithms in a computer based on discarding gram lists and combining correlated lists.