LLM Query Response Pipeline With Chunk Ranking for External Knowledge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) struggle to provide accurate responses to queries related to information not included in their training dataset, such as enterprise-specific knowledge, necessitating a technique like retrieval augmented generation (RAG) to leverage external knowledge bases.
Innovation Solution
An LLM-based RAG system that generates synthetic information from a query, modifies it, retrieves relevant document chunks using an information retrieval system, ranks these chunks with another LLM, selects a subset, and generates a response based on the ranked chunks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a standard LLM is used to generate responses, then the system is simple and fast, but it cannot provide accurate answers for information outside its training dataset
Solution Approach 1:
The patent introduces an information retrieval system as an intermediary component between the query and the LLM. This retrieval system fetches relevant external knowledge from a corpus of documents, which is then combined with the query and provided to the LLM. This mediator enables the LLM to access information outside its training data without requiring retraining, thereby improving accuracy while maintaining relative system simplicity.
Solution Approach 2:
The system segments the response generation process into distinct stages: (1) query processing and modification by the first LLM, (2) information retrieval from external corpus, (3) ranking and selection of retrieved chunks by the second LLM, and (4) final response generation by the third LLM. This segmentation allows each component to specialize in specific tasks, improving overall reliability while managing complexity through modular design.
2Reliability
If retrieval augmented generation is used to access external knowledge, then accuracy for domain-specific queries improves, but system complexity and processing time increase
Solution Approach 1:
The system performs preliminary actions by pre-processing the external corpus into searchable chunks and maintaining an indexed structure before queries are submitted. The retrieval system is pre-configured with the corpus structure, enabling faster retrieval during actual query processing. This preliminary preparation reduces the time penalty associated with accessing external knowledge during live operations.
Solution Approach 2:
The system extracts only the most relevant portions of external knowledge (specific chunks from the corpus) that are necessary to answer each query, rather than processing or retrieving entire documents. The second LLM ranks retrieved chunks and selects only the most relevant subset for final response generation, reducing processing time while maintaining accuracy.
3Reliability
If multiple LLMs are used for query modification, ranking, and response generation, then response quality improves, but computational resources and system complexity increase
Solution Approach 1:
The patent employs the same type of LLM technology for all three functions (query modification, ranking, and response generation), leveraging the universal capabilities of LLMs across different tasks. Each LLM is prompted to perform its specific function, but they all utilize the same underlying technology platform, which reduces infrastructure complexity compared to using specialized models for each task.
Solution Approach 2:
The system segments the complex task of knowledge-based response generation into three distinct functional stages, each handled by a specialized LLM component. This segmentation allows for optimized resource allocation and enables parallel processing possibilities, managing computational complexity through functional decomposition while maintaining high response quality through specialized processing at each stage.
Data Source
AI summary
Systems and method for generating a response to a query. The method includes using a first large language model (LLM) to generate synthetic information related to a query; generating an amended query based on the synthetic information related to the query; using an information retrieval system to retrieve, from a plurality of chunks, a set of chunks that are relevant to the amended query, wherein each chunk of the plurality of chunks is all or a portion of a document in a corpus of documents; using a second LLM to rank the set of chunks based on a relevance to the query; selecting a subset of chunks from the set of chunks based on the ranking; and using a third LLM to generate a response to the query based on the subset of chunks.