LLM Chunk Re-Ranking for Enterprise Query Response Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) struggle to provide accurate responses to queries related to information not included in their training dataset, such as enterprise-specific knowledge, necessitating a more efficient and automated method for generating responses using retrieval augmented generation (RAG) systems.
Innovation Solution
A system utilizing a re-ranker LLM to rank relevant document chunks based on relevance to a query via chain-of-thought prompting, followed by a generation LLM to generate a response using a subset of these chunks, enhanced by a query modification LLM to refine queries and an information retrieval system to identify relevant chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a standard LLM is used to generate responses to queries, then the system is simple and fast, but the accuracy deteriorates when the query relates to information not in the LLM's training dataset
Solution Approach 1:
An information retrieval system acts as an intermediary between the query and the LLM. The retrieval system first identifies relevant document chunks from an external knowledge base, which are then provided to the LLM as additional context. This mediator enables the LLM to access information beyond its training data, improving accuracy for domain-specific or internal enterprise knowledge queries.
Solution Approach 2:
The system performs preliminary information retrieval before the LLM generates the response. Relevant document chunks are identified and retrieved in advance, and the LLM is given these pre-selected materials to work with. This preliminary action ensures the LLM has access to accurate, relevant information before attempting to generate the final response.
2Reliability
If an information retrieval system is added to provide external knowledge, then the accuracy improves, but the response time increases
Solution Approach 1:
The external knowledge base is segmented into smaller document chunks, which are then individually processed and ranked. This segmentation allows the system to retrieve only the most relevant portions rather than processing entire documents, reducing the time required while maintaining accuracy.
Solution Approach 2:
The system uses a re-ranker LLM to change the ranking parameter of retrieved chunks based on their relevance to the query. By adjusting the ranking order dynamically according to query-specific relevance, the system quickly identifies and prioritizes the most important information, reducing response time while maintaining high accuracy.
3Measurement precision
If a re-ranker LLM with chain-of-thought prompting is used to rank chunks, then the relevance accuracy improves, but the computational resources required increase
Solution Approach 1:
Instead of applying chain-of-thought prompting to all retrieved chunks, the system applies it selectively to the top-ranked chunks or uses a simplified ranking approach for less critical chunks. This partial application reduces computational overhead while maintaining high relevance accuracy for the most important information.
Solution Approach 2:
The ranking process is segmented into multiple stages: an initial ranking pass using standard methods, followed by a more computationally intensive chain-of-thought re-ranking pass applied only to the top candidates. This segmented approach distributes computational load efficiently, achieving high accuracy without excessive resource consumption across the entire dataset.
Data Source
AI summary
Methods and systems for generating a response to a query. The methods comprising: receiving a set of chunks that are relevant to the query, the set of chunks from a plurality of chunks generated from documents in a corpus of documents; using a re-ranker LLM to rank the set of chunks that are relevant to the query based on a relevance to the query via chain-of-thought prompting; selecting a subset of chunks from the set of chunks based on the ranking; and using a generation LLM to generate the response to the query based on the subset of chunks.