Search Index Stopword Coalescing for Phrase Query Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines face performance bottlenecks when handling phrase queries due to frequent stopwords, which occupy significant disk space and impact system performance, with existing solutions like skiplists and next-word indexing either increasing index complexity or adding millions of unique words.
Innovation Solution
Coalescing stopwords with the first letter or prefix of the next word to create specialized internal tokens, reducing the number of unique words and simplifying the indexing process, while maintaining positional information for efficient querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If stopwords are coalesced with the entire next word (next-word indexing), then phrase query performance is improved, but the number of unique words in the index increases by millions
Solution Approach 1:
coalescing the stopword and a prefix, such as the first letter, of the next word to create a specialized internal token
Solution Approach 2:
The patent applies different levels of coalescing based on local needs - only the first letter of the next word is coalesced with the stopword, providing just enough distinction to maintain query accuracy while minimizing index growth
2Speed
If skiplists are used for selective access of index postings, then phrase query speed is improved, but index complexity and index size increase
Solution Approach 1:
The patent changes the parameter of how stopword data is represented in the index - instead of using complex skiplist structures, it transforms stopwords into compact coalesced tokens that can be efficiently stored and retrieved using standard index structures
3Ease of manufacture
If stopwords are kept as separate entries in the index, then indexing simplicity is maintained, but disk performance and system memory usage are impacted
Solution Approach 1:
The patent merges the stopword with a prefix of the next word into a single coalesced token, reducing the total volume of data that must be stored and retrieved from disk while maintaining the ability to accurately process phrase queries
Data Source
Figure 1
Figure 2~4
Figure 5
AI summary
Phrases in a corpus of documents including stopwords are found using a data processor arranged to execute phrase queries. Memory stores an index structure which maps entries in the index structure to documents in the corpus. Entries in the index structure represent words and other entries represent stopwords found in the corpus coalesced with prefixes of respective adjacent words adjacent to the stopwords. The prefixes comprise one or more leading characters of the respective adjacent words. A query processor forms a modified query by substituting a stopword with a search token representing the stopword coalesced with a prefix of the next word in the query. The processor executes the modified query. Also, index structures including coalesced stopwords are created and maintained.