Tiered Sharded Phrase Posting Lists for Index Servers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional information retrieval systems index documents based on individual terms rather than concepts, leading to limitations in capturing topical relationships between phrases and requiring extensive computational resources, as well as inefficient query processing due to the need to access multiple index servers.
Innovation Solution
An information retrieval system that indexes and searches documents using phrases, employing tiered and sharded phrase posting lists to optimize query processing by decomposing queries into phrase trees, scheduling searches, and maintaining indexes across multiple servers to minimize communication and processing costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If documents are indexed based on individual terms, then the index can cover all possible search queries, but the system requires extensive computational resources and memory to process queries efficiently
Solution Approach 1:
The patent segments the index into multiple tiers based on phrase frequency and importance. Tier 1 contains high-frequency phrases that are accessed most often, while lower tiers contain less frequent phrases. This segmentation allows the system to optimize query processing by focusing computational resources on the most frequently accessed phrases, reducing overall computational overhead while maintaining comprehensive query coverage.
Solution Approach 2:
The patent applies local quality by assigning different storage and processing characteristics to different parts of the index based on their access patterns. High-frequency phrases in Tier 1 receive optimized storage and faster access mechanisms, while lower-frequency phrases in lower tiers use more compressed or less optimized storage. This differential treatment reduces overall computational resources while maintaining adaptability.
2Quantity of substance
If the index is distributed across multiple servers, then the system can handle larger document collections, but query processing requires accessing multiple servers increasing communication overhead
Solution Approach 1:
The patent segments the distributed index into hierarchical tiers across multiple servers. Each server hosts specific tiers, and the system routes queries to appropriate servers based on the query phrases and their tier assignments. This segmentation reduces the number of servers that need to be accessed for each query, minimizing communication overhead while maintaining the ability to handle large document collections.
Solution Approach 2:
The patent introduces a front-end server that acts as an intermediary between clients and the distributed index servers. The front-end server receives queries, determines which tiers and servers are relevant, and coordinates access to the appropriate servers. This intermediary optimizes query routing and reduces unnecessary server communications, decreasing query processing time while supporting large-scale document collections.
3Adaptability or versatility
If all phrase posting lists are stored on all index servers, then any server can handle any query, but the storage requirements for each server become prohibitively large
Solution Approach 1:
The patent segments phrase posting lists into hierarchical tiers and distributes them across multiple servers. Each server stores only the tiers it is responsible for, rather than all tiers. This segmentation reduces the storage requirements on each individual server while maintaining system-wide adaptability through the hierarchical structure that allows queries to be routed to the appropriate servers based on their tier assignments.
4Loss of information
If the system indexes all possible phrases, then it can capture all topical relationships, but the memory requirements to store all phrase posting lists become unmanageable
Solution Approach 1:
The patent applies local quality by differentiating between high-frequency and low-frequency phrases, and between important and less important topical relationships. High-frequency phrases that represent common topical relationships are stored in accessible tiers with optimized memory usage, while low-frequency phrases are stored in lower tiers with more compressed storage or on-demand generation. This approach captures essential topical relationships while managing memory requirements through differential treatment of different phrase types.
Data Source
AI summary
An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are extracted from the document collection. Documents are the indexed according to their included phrases, using phrase posting lists. The phrase posting lists are stored in an cluster of index servers. The phrase posting lists can be tiered into groups, and sharded into partitions. Phrases in a query are identified based on possible phrasifications. A query schedule based on the phrases is created from the phrases, and then optimized to reduce query processing and communication costs. The execution of the query schedule is managed to further reduce or eliminate query processing operations at various ones of the index servers.