Long-Document RAG Using Segment Ranking for Context Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face limitations due to context window size constraints, which restrict the number of tokens they can process, leading to insufficient context information and increased computational costs when handling longer documents.
Innovation Solution
A method that divides long documents into segments, assigns relevance scores using a language model, selects the top-k segments, and combines them into a virtual document within the context window size limit, enabling effective response generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the context window size is increased to handle longer documents, then the ability to process more context information is improved, but the computational costs increase dramatically
Solution Approach 1:
The patent divides the long source document into multiple segments or chunks, each fitting within the LLM's context window. These segments are processed separately and then aggregated to form the complete answer, enabling handling of documents longer than the context window without increasing the window size itself.
Solution Approach 2:
The patent extracts only the most relevant segments from the source document based on their relationship to the query. By identifying and extracting pertinent portions rather than processing the entire document, the system maintains efficient computational costs while providing adequate context information.
2Reliability
If the context window size is increased to provide more context information, then the quality of response generation is improved, but the computational resources required increase
Solution Approach 1:
The patent applies different processing strategies to different segments of the document based on their relevance to the query. High-relevance segments receive more detailed processing and are included in the final context, while lower-relevance segments are summarized or excluded, optimizing the balance between response quality and computational resource usage.
Solution Approach 2:
The patent processes only the necessary portions of the document required to answer the query adequately, rather than processing the entire document. This partial action approach maintains response quality by including all relevant information while avoiding the computational overhead of processing unnecessary content.
3Loss of information
If the entire source document is processed to ensure complete information coverage, then the comprehensiveness of the answer is improved, but the processing time and computational load increase
Solution Approach 1:
The patent performs preliminary processing of the source document by dividing it into segments and pre-assessing their relevance to the query before generating the final answer. This preliminary action includes creating embeddings or summaries of segments that can be quickly evaluated, reducing the time required for comprehensive information coverage during actual query processing.
Solution Approach 2:
The patent introduces an intermediary retrieval mechanism that acts as a bridge between the query and the source document. This intermediary system (such as a retrieval-augmented generation component) pre-processes and indexes document segments, enabling fast retrieval of relevant information without processing the entire document during query response generation.
Data Source
AI summary
A computer-implemented method comprising: receiving, as input, a query and a source document intended for a content-grounded question-answering or multi-turn conversation task by a specified large language model (LLM) which has a context window size limit, wherein the source document has a size which exceeds the context window size limit; dividing the source document into a plurality of segments; applying a language model to each of the segments, to assign to each of the segments a relevance score; selecting the k-top segments having the highest the relevance scores; combining the selected k-top segments into a virtual document having a size which complies with the context window size limit; and feeding the virtual document as input to the specified LLM, to generate a response that is grounded in the content of the virtual document.