Multi-stage Query Processing with Tokenspace Repository
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information retrieval systems face challenges in providing relevant query results due to the degradation of search performance caused by sophisticated scoring techniques that require additional processing overhead, such as partial document reconstruction for context determination.
Innovation Solution
A multi-stage query processing system and method using a tokenspace repository, which enables incremental document reconstruction through a multi-tiered mapping scheme, allowing for multiple passes on a user query to improve document relevancy scores and presentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sophisticated scoring techniques are introduced to improve query relevance, then measurement precision is improved, but device complexity increases due to additional processing overhead
Solution Approach 1:
The patent segments the query processing into multiple stages: initial scoring using compressed tokenspace, then selective reconstruction of only candidate documents that meet threshold criteria. This segmentation allows sophisticated scoring to be applied only where needed, reducing overall processing overhead while maintaining measurement precision for relevant documents.
Solution Approach 2:
The patent implements partial document reconstruction rather than full reconstruction for all documents. Only candidate documents that meet certain scoring thresholds undergo reconstruction to determine context, applying the sophisticated scoring technique partially to avoid excessive processing overhead while still improving query relevance for the most promising candidates.
2Measurement precision
If partial document reconstruction is performed to determine context of query terms, then measurement precision is improved, but loss of time increases due to additional processing
Solution Approach 1:
The patent performs preliminary scoring using the compressed tokenspace representation before undertaking time-consuming document reconstruction. This preliminary action filters out documents that cannot possibly meet relevance thresholds, so context determination through reconstruction is only performed on a small subset of candidate documents, minimizing time loss while maintaining precision.
Solution Approach 2:
The patent applies different processing qualities to different documents based on their initial scoring. High-scoring candidates receive full context determination through reconstruction, while lower-scoring documents receive only compressed-space scoring. This local quality approach ensures measurement precision where it matters most while reducing overall processing time.
Data Source
AI summary
A multi-stage query processing system and method enables multi-stage query scoring, including “snippet” generation, through incremental document reconstruction facilitated by a multi-tiered mapping scheme. At one or more stages of a multi-stage query processing system a set of relevancy scores are used to select a subset of documents for presentation as an ordered list to a user. The set of relevancy scores can be derived in part from one or more sets of relevancy scores determined in prior stages of the multi-stage query processing system. In some embodiments, the multi-stage query processing system is capable of executing one or more passes on a user query, and using information from each pass to expand the user query for use in a subsequent pass to improve the relevancy of documents in the ordered list.