Phrase Indexing Using Tiered Sharding for Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional information retrieval systems index documents based on individual terms rather than concepts, leading to limitations in capturing topical relationships between phrases and requiring extensive computational resources, as well as inefficient query processing due to the need to access multiple index servers.
Innovation Solution
An information retrieval system that indexes and searches documents using phrases, decomposing queries into phrase trees, optimizing query execution, and utilizing tiered and sharded index servers to minimize inter-server communication and improve query processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If documents are indexed based on individual terms, then the indexing process is simpler and requires fewer computational resources, but the system cannot capture topical relationships between phrases and returns less accurate search results
Solution Approach 1:
The patent segments the index into multiple tiers: individual term indices and phrase indices. This segmentation allows the system to maintain both simple term-based indexing and complex phrase-based indexing without requiring the entire system to be overly complex. The phrase index is further segmented into different phrase lengths (bigrams, trigrams, etc.), enabling selective use of phrase indexing based on query requirements.
Solution Approach 2:
The patent adds a new dimension to the traditional single-level term index by introducing phrase-level indexing as an additional layer. This dimensional expansion allows the system to capture topical relationships between words while maintaining the original term-based indexing structure. The multi-dimensional index structure enables queries to leverage both term matches and phrase matches depending on what is most appropriate.
2Measurement precision
If a comprehensive phrase index is created to capture all possible phrases, then topical relationships are captured accurately, but the computational and memory requirements become prohibitively large
Solution Approach 1:
The patent applies local quality by creating phrase indices selectively rather than universally. Not all possible phrases are indexed - only those that meet certain criteria such as frequency thresholds, length constraints, and relevance to the document collection. This selective approach allows the system to capture important topical relationships while avoiding the memory overhead of indexing every possible word combination.
Solution Approach 2:
The patent implements partial action by indexing only a subset of all possible phrases - specifically, phrases that are statistically significant and topically relevant. Rather than creating a complete phrase index for all possible word combinations, the system creates phrase indices for the most important phrases, achieving sufficient topical relationship capture without excessive memory consumption.
3Measurement precision
If queries are processed by accessing multiple index servers, then comprehensive search results are obtained, but query processing time increases due to extensive inter-server communication
Solution Approach 1:
The patent segments the index servers into multiple tiers, where each tier handles specific types of queries or specific portions of the index. This segmentation allows queries to be routed to the appropriate tier without requiring access to all index servers. The sharding of index data across multiple servers within each tier further reduces communication overhead by localizing data access.
Solution Approach 2:
The patent implements preliminary action by pre-computing and caching phrase posting lists and their intersections. When phrases are frequently queried together, their intersection results are pre-computed and stored, so that subsequent queries can retrieve results without performing real-time intersections across multiple index servers. This eliminates redundant inter-server communication for common query patterns.
4Measurement precision
If phrase indexing is implemented to capture concepts, then search relevance improves, but the device complexity increases due to tiered and sharded index server architecture
Solution Approach 1:
The patent segments the complex index server architecture into manageable tiers and shards, where each component has a specific, simplified function. This segmentation makes the overall complex system easier to understand, maintain, and scale. Each tier handles a specific aspect of phrase indexing, and each shard manages a portion of the data, reducing the complexity burden on any single component.
Solution Approach 2:
The patent implements multi-functionality by designing the tiered and sharded index server architecture to handle multiple types of queries (term queries, phrase queries, partial match queries) through a unified framework. This universal architecture reduces device complexity by avoiding the need for separate specialized systems for different query types, as the same infrastructure supports all query modalities.
Data Source
AI summary
An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are extracted from the document collection. Documents are the indexed according to their included phrases, using phrase posting lists. The phrase posting lists are stored in an cluster of index servers. The phrase posting lists can be tiered into groups, and sharded into partitions. Phrases in a query are identified based on possible phrasifications. A query schedule based on the phrases is created from the phrases, and then optimized to reduce query processing and communication costs. The execution of the query schedule is managed to further reduce or eliminate query processing operations at various ones of the index servers.