RAG Document Ranking for Accurate Context Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models generate outdated, incorrect, or hallucinatory answers due to outdated training data, and current retrieval-augmented generation systems face challenges in assessing reference document quality, managing context size, and optimizing document selection and ordering, leading to biased and redundant results.
Innovation Solution
Implement a document selection LLM with a document ranking and preference optimization layer to enhance document selection and ordering, using a retrieval optimization engine to train the system with a ranking LLM and field LLM, optimizing reference document lists for improved relevance and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If retrieval-augmented generation systems use external data to improve answer accuracy, then the relevance and accuracy of answers improve, but the system complexity and computational overhead increase
Solution Approach 1:
The system segments the document selection process into multiple stages: initial document retrieval, quality assessment by the LLM, relevance ranking, and final selection. This segmentation allows each component to focus on a specific aspect of document evaluation, improving overall answer accuracy while managing system complexity through modular design
Solution Approach 2:
The patent introduces an intermediary LLM-based document selection system that acts as a mediator between the external data corpus and the final answer generation. This intermediary assesses and ranks documents before they are used for answer generation, ensuring higher quality inputs without requiring complete reprocessing of all external data
2Reliability
If the system processes more reference documents to improve answer quality, then the accuracy improves, but the processing time and computational resources increase
Solution Approach 1:
The system applies partial action by selecting only a subset of the most relevant documents from the retrieved corpus rather than processing all available documents. The LLM assesses document quality and selects the top-k most relevant documents, achieving high answer quality without the computational cost of processing the entire document set
Solution Approach 2:
The patent implements preliminary action through pre-assessment and pre-ranking of documents before they are used for answer generation. The document selection LLM evaluates and ranks documents in advance, so that when answer generation is needed, only pre-validated high-quality documents are processed, significantly reducing real-time processing time
3Reliability
If the document selection process is optimized for relevance, then the answer relevance improves, but the risk of selection bias increases
Solution Approach 1:
The system incorporates feedback mechanisms where the LLM evaluates not only the relevance of individual documents but also the diversity and representativeness of the selected document set. This feedback loop allows the system to adjust selections to maintain relevance while mitigating bias by ensuring balanced representation from different sources and perspectives
Solution Approach 2:
The patent changes the evaluation parameters from simple relevance scoring to a multi-dimensional assessment including document quality, relevance, diversity, and source credibility. By adjusting these parameters in the document selection process, the system achieves high answer relevance while reducing selection bias through more nuanced evaluation criteria
4Reliability
If the system manages larger context sizes to include more external data, then the comprehensiveness of answers improves, but the computational load and memory requirements increase
Solution Approach 1:
The system extracts only the essential and most relevant information from external documents rather than processing complete documents. The LLM identifies and extracts key passages, facts, and data points that are necessary for answering the query, reducing context size and computational load while maintaining answer comprehensiveness
Solution Approach 2:
The patent applies partial action by including only the necessary portion of external data in the context window. Instead of loading complete documents, the system selectively includes relevant excerpts and summaries, achieving comprehensive answers with reduced memory requirements and lower computational energy consumption
Data Source
AI summary
A method includes applying a document ranking layer of a document selection large language model (LLM) to a document list including multiple reference documents to obtain a ranked document list. The method further includes selecting a subset of reference documents from the ranked document list and processing a user prompt and the document subset by a field LLM to generate an answer. The method further includes ranking the answer with an answer score by a ranking LLM. The method further includes ranking the document subset by the ranking LLM to obtain a ranked document subset. The method further includes calculating a loss function of a preference optimization layer of the document selection LLM based on the answer score and updating at least one training parameter of a foundation model of the document selection LLM based on the loss function of the preference optimization layer.