Fuzzy Full Text Search Using NVC Tree Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current full text search systems, such as SQLite and Lucene, face performance issues and lack fault-tolerant capabilities in handling misspelled search queries, with Lucene's two-step fuzzy search process being inefficient and SQLite not supporting fuzzy searching at all.
Innovation Solution
A fuzzy full text search method utilizing an inverted token Next Valid Character (NVC) tree, which allows for a single-pass traversal to identify relevant document sets, using the same index structure for both full text and fuzzy searches by adjusting the error value from zero for exact searches to greater than zero for fuzzy searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Lucene performs fuzzy search using a two-step process computing edit distance, then fuzzy search capability is achieved, but severe performance problems occur
Solution Approach 1:
The patent pre-computes and stores Next Valid Character (NVC) information in the inverted index during indexing. This preliminary action allows the search phase to directly traverse the pre-organized NVC tree without computing edit distances at query time, resolving the performance issue while maintaining fuzzy search capability
Solution Approach 2:
The patent segments the search process into character-level traversal through the NVC tree rather than computing full edit distances. By breaking down token matching into incremental character comparisons guided by NVC information, the system achieves fuzzy matching without the severe performance penalty of traditional two-step edit distance computation
2Productivity
If SQLite FTS system uses inverted indexes for efficient full text search, then exact match performance is improved, but fault-tolerant searching capability is lost
Solution Approach 1:
The patent makes the inverted index structure universal by enabling it to perform both exact match search (when error value is zero) and fuzzy search (when error value is greater than zero). The same NVC tree structure supports multiple search modes, eliminating the need for separate fuzzy search mechanisms and maintaining both performance and fault tolerance
Solution Approach 2:
The patent introduces an error value parameter that controls the search behavior. By changing this parameter from zero to a positive value, the system transitions from exact match to fuzzy match mode. This parameter change allows a single inverted index to adapt to different search requirements without sacrificing performance
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and system for fuzzy full text search is disclosed. The system includes an inverted index where tokens are organized in a Next Valid Character (NVC) tree. Each path through the tree defines a token. The document lists for the tokens are stored at leaf nodes. When performing a fuzzy full text search, the system uses an edit distance greater than zero. After receiving search tokens, the system traverses the NVC tree to generate document error lists. The system then compares the error lists to identify relevant document identifiers and provides a result set.