Retrieval-Augmented Q&A for Citable LLM Answers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) face challenges in handling complex data types like tables and visuals, incorporating human feedback, and providing recommendations beyond answering questions, with issues in storage and memorization of facts leading to inaccuracies and lack of citable outputs.
Innovation Solution
A retrieval-based augmentation method segments documents into latent semantic embeddings, incorporating user feedback and complex data types, and generates responses with embedded document segments, allowing LLMs to provide accurate answers and recommendations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs store facts in their neural network weights, then they can recall information from training data, but the storage mechanism is not understood and factual errors (hallucinations) occur
Solution Approach 1:
The patent introduces a retrieval system as an intermediary between the user query and the LLM. Instead of relying solely on the LLM's internal weights, the system first retrieves relevant document segments using semantic search, then feeds these retrieved segments to the LLM as context. This intermediary retrieval step provides verifiable sources for the LLM's responses, reducing hallucinations while maintaining recall accuracy.
2Adaptability or versatility
If LLMs are trained on finite text corpora, then they can answer questions from training data, but they cannot answer questions on documents not present in the training data
Solution Approach 1:
The patent performs preliminary indexing and embedding of document segments before they are needed for answering queries. By pre-processing the document corpus into searchable embeddings and storing them in a retrieval database, the system prepares the knowledge base in advance. When a new query arrives, the system can quickly retrieve relevant segments without requiring retraining, enabling the LLM to answer questions from documents it has never seen during training.
3Reliability
If information in documents changes, then the LLM must be retrained or fine-tuned on new documents, but this is expensive and time-consuming
Solution Approach 1:
The patent implements a dynamic retrieval system where the knowledge base can be updated without retraining the LLM. When documents change, only the affected document segments need to be re-indexed and re-stored in the retrieval database. The LLM itself remains unchanged, but the retrieval system dynamically adapts to new information by updating its embedded document segments. This dynamic approach allows the system to stay current with minimal computational overhead.
4Adaptability or versatility
If complex data types like tables and visuals are included in documents, then the Q&A system can handle more information types, but it becomes difficult to incorporate them as text in the LLM prompt
Solution Approach 1:
The patent extracts and separates different data types from complex documents during the segmentation and embedding process. Tables, visuals, and other non-text elements are extracted as distinct components and converted into text-based representations or descriptions that can be incorporated into the LLM prompt. This extraction approach allows the system to handle complex data types by converting them into a format suitable for text-based LLM processing, maintaining versatility while managing complexity.
5Productivity
If the LLM provides answers based on stored weights, then responses are generated quickly, but citations for the output cannot be easily provided
Solution Approach 1:
The patent introduces a retrieval system as an intermediary that provides citable sources for LLM outputs. The retrieval component identifies and returns specific document segments that support the LLM's answer, along with metadata such as source identifiers and locations. This intermediary step maintains the speed of LLM generation while adding traceability, as the retrieved segments serve as direct citations that can be referenced and verified by users.
Data Source
AI summary
Large language models (LLMs) are versatile in responding to user questions on a wide variety of topics. However, LLMs suffer from several drawbacks, such as hallucinations, incomplete information, and inability to cite original sources of information. Disclosed herein are systems and methods for using an LLM in a restricted manner to respond to queries regarding document corpora, e.g., documents related to a set of products, such that the impact of these drawbacks is minimized. Information retrieval is coupled with LLMs to build a question and answer (Q&A) system on the text corpora. Complex retrieved information, incorporating human feedback, and recommendations in the Q&A system are provided.