Top-k Query Processing with Conditional Document Skipping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The efficiency of top-k query processing in search engines is compromised as the size of the document collection grows larger, and current methods struggle with longer user queries, leading to increased computational overhead during the first phase of the search process.
Innovation Solution
Dynamic pruning algorithms are employed to reduce the number of evaluated documents by using a conditional skip iterator and TREAP data structure, which skips documents that cannot belong to the final result list based on query term contributions and upper bounds, improving the efficiency of top-k query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional top-k query processing evaluates all matching documents, then result accuracy is maintained, but computational overhead increases significantly with larger document collections
Solution Approach 1:
The patent segments the evaluation process by dividing documents into evaluated and skipped groups based on conditional criteria. The conditional skip iterator segments the document evaluation process, allowing only certain documents to be fully evaluated while others are skipped based on upper bound calculations, thus maintaining accuracy for relevant documents while improving efficiency.
Solution Approach 2:
The patent applies partial action by evaluating only the necessary subset of documents rather than all matching documents. The conditional skip mechanism performs partial evaluation by skipping documents that cannot contribute to the top-k results, reducing computational overhead while maintaining result accuracy through selective full evaluation.
2Speed
If dynamic pruning skips more documents, then query processing speed improves, but the risk of missing relevant documents increases
Solution Approach 1:
The patent performs preliminary action by calculating upper bounds for query term contributions before final scoring. This preliminary estimation allows the system to identify and skip documents that cannot exceed the current top-k threshold, while ensuring that documents with potential to rank in top-k undergo full evaluation, thus maintaining reliability.
Solution Approach 2:
The patent implements feedback through the conditional skip iterator that continuously monitors the current top-k threshold and adjusts document evaluation decisions accordingly. The system provides feedback by comparing upper bound estimates against the current threshold, dynamically adjusting which documents to evaluate or skip, ensuring result completeness while optimizing speed.
3Measurement precision
If all query terms are processed for every document, then scoring accuracy is maintained, but computational complexity increases with longer queries
Solution Approach 1:
The patent extracts and processes only the necessary query terms for each document based on the conditional skip criteria. By taking out only the relevant term contributions that can affect the top-k results, the system maintains scoring accuracy for evaluated documents while reducing computational complexity through selective term processing.
Data Source
AI summary
Methods and apparatus for performing top-k query processing include pruning a list of documents to identify a subset of the list of documents, where pruning includes, for other query terms in the set of query terms, skipping a document in the list of documents based, at least in part, on the contribution of the query term to the score of the corresponding document and the term upper bound for each other query term, in the set of query terms, that matches the document.