Generative Search Prompt Granularity for Query Complexity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative language models face challenges in accessing expert knowledge without retraining, incur high costs due to increased parameter numbers, and generate inaccurate content (hallucinations), while retrieval-augmented models require long prompts that exceed positional embedding limits and incur API fees proportional to token length.
Innovation Solution
An inference method and system that configures prompts for generative language models based on a ranking of length-by-length phrases, adjusting rankings using a granularity coefficient that reflects query complexity, and employs a transformer encoder model to predict this coefficient, ensuring relevance and conciseness based on query complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If retrieved documents are pasted to form a long prompt for generative search, then the language model can access sufficient knowledge, but the API cost increases proportionally to the number of tokens and the prompt exceeds the maximum length for inference
Solution Approach 1:
The patent extracts only the essential query-related parts from retrieved documents using a phrase retriever, rather than including entire documents. This extraction process selects specific phrases that are most relevant to the query, significantly reducing prompt length while maintaining knowledge coverage.
Solution Approach 2:
The patent applies different selection criteria to different parts of the retrieved documents based on their relevance to the query. High-relevance regions are selected with higher priority, while less relevant regions are excluded. This local quality approach ensures that the most important information is included while minimizing unnecessary content.
2Quantity of substance
If a phrase retriever is used to extract only query-related parts, then the prompt length is reduced, but the accuracy of selecting relevant information decreases
Solution Approach 1:
The patent dynamically adjusts the selection of phrases based on the query characteristics and the importance score calculated for each phrase. The selection process is not static but adapts to the specific query and document context, allowing the system to optimize between prompt length and relevance accuracy for each individual search request.
Solution Approach 2:
The patent uses importance scores as a feedback mechanism to evaluate and select phrases. The importance score is calculated based on the relationship between the query and each phrase, providing feedback that guides the selection process. This feedback loop ensures that the most relevant phrases are selected while maintaining control over prompt length.
3Device complexity
If the ranking of phrases is based only on relevance between query and phrase, then the selection process is simple, but the prompt may include too much or too little context depending on query complexity
Solution Approach 1:
The patent changes the ranking parameter from simple relevance score to an adjusted importance score that incorporates both relevance and phrase length. This parameter change allows the system to adapt to different query complexities by controlling the amount of context included in the prompt, balancing simplicity with adaptability.
Data Source
AI summary
Disclosed is a method and system for readjusting granularity of search results according to complexity of a query for generative search. An inference method may include extracting a length-by-length phrase from a document retrieved for a query of a user; determining a ranking of the length-by-length phrase in consideration of complexity of the query; and configuring a prompt for input to a generative language model based on the determined ranking. Here, the ranking of the length-by-length phrase may be determined such that a relatively long phrase has a higher ranking according to an increase in the complexity of the query.