Full Text Search Index Partitioning for Navigation Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Full text search systems face inefficiencies when handling queries with non-selective tokens, particularly those found in most documents, leading to prolonged processing times and increased storage needs, as they often require iterating over large document sets, even when only a small number of documents match the query.
Innovation Solution
The method partitions the full text search index based on frequently used terms, mapping them to dedicated indexes where the term itself is not stored, and uses an additional relation to determine which indexes to query, reducing storage and query time by avoiding explicit document lists and optimizing searches for non-selective tokens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the FTS system stores explicit document lists for all tokens in the inverted index, then complete search coverage is achieved, but storage space consumption increases significantly
Solution Approach 1:
The patent segments the inverted index into multiple parts: a compressed index structure that stores only essential token information, and separate document lists that are loaded on-demand during search operations. This segmentation allows the system to maintain complete search coverage while significantly reducing the amount of data that must be continuously stored in memory.
Solution Approach 2:
The patent implements a nested structure where the inverted index contains references to document lists, which in turn contain the actual document identifiers. This nested organization allows the system to store compact index structures while maintaining the ability to access complete document information when needed, effectively nesting the essential data structure within the full document information.
2Reliability
If the FTS system iterates over large document sets to generate result sets, then accurate search results are produced, but query processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing compressed index structures that capture the essential relationships between tokens and documents. During query processing, these pre-computed structures allow the system to quickly identify candidate documents without iterating over entire document sets, significantly reducing query processing time while maintaining search accuracy.
Solution Approach 2:
The patent extracts only the essential information needed for search operations from the complete document sets and stores it in the inverted index. During query processing, the system extracts candidate document identifiers from the compressed index structure without needing to iterate over the full document sets, thereby reducing processing time while preserving search accuracy.
3Reliability
If the FTS system includes stop-words like 'the' in the inverted index, then comprehensive token coverage is achieved, but query performance degrades due to large document set comparisons
Solution Approach 1:
The patent applies local quality by treating different tokens differently in the inverted index. Stop-words and common tokens are stored in a compressed format with optimized data structures, while less common tokens maintain their traditional representations. This localized optimization allows the system to maintain comprehensive token coverage while improving query performance for frequently occurring tokens.
Solution Approach 2:
The patent changes the parameters of token storage by using compressed data structures and optimized encoding schemes for stop-words and common tokens. This parameter change allows the system to maintain the presence of these tokens in the inverted index for comprehensive coverage while reducing the storage requirements and improving query processing efficiency through compact representations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and system for full text search for use during destination selection with a navigation system is disclosed. The method partitions a full text search index based on frequently used terms. These frequently used terms are then mapped to dedicated full text search indexes, where each document contains this frequently used term but the term itself is not stored in the index. A relation maps the frequently used terms to the inverted indexes.