Knowledge Graph Retrieval for Context-Coherent LLM Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional retrieval augmented generation (RAG) systems in chatbots rely on semantic similarity, leading to superficially relevant or irrelevant responses due to the reliance on document selection that may not be contextually appropriate, and often fail to include necessary information, exacerbating the issue of hallucinations, particularly in domains requiring accuracy like legal and healthcare.
Innovation Solution
A method utilizing a knowledge graph with hierarchical and interconnected document structures, combined with embedding models, to dynamically retrieve and cluster documents based on contextual relationships, ensuring that responses are grounded in a contextually rich and coherent subset of information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If semantic similarity is used to select documents for RAG, then retrieval speed is improved, but document selection accuracy deteriorates leading to superficially relevant but contextually incorrect information
Solution Approach 1:
The patent introduces an intermediary verification mechanism that checks whether retrieved documents contain the specific answer to the query, not just semantic similarity. This intermediary step filters out superficially relevant but contextually incorrect documents (like Kiger mustangs for 1977 Mustang queries) while maintaining retrieval efficiency.
Solution Approach 2:
The system implements feedback by evaluating whether the retrieved documents actually contain the answer to the user's query. This feedback loop allows the system to adjust document selection based on whether the documents provide genuine contextual relevance, preventing hallucinations from contextually inappropriate information.
2Quantity of substance
If semantic similarity threshold is lowered to include more documents, then information completeness is improved, but relevance quality deteriorates including irrelevant information
Solution Approach 1:
An intermediary verification step is introduced that checks each retrieved document to determine if it actually contains the answer to the query. This mediator filters out irrelevant documents even if they were retrieved due to low similarity thresholds, ensuring that quantity does not compromise quality.
Solution Approach 2:
The patent applies local quality by evaluating each document individually for contextual relevance rather than treating all retrieved documents uniformly. Documents are assessed based on their specific content's relevance to the query, allowing the system to include multiple diverse documents while excluding those that don't provide genuine answer content.
3Loss of information
If conventional RAG provides more context documents to LLM, then response coverage is improved, but hallucination risk increases due to inclusion of irrelevant information
Solution Approach 1:
An intermediary verification mechanism is introduced that validates whether each retrieved document actually contains the answer to the query before including it in the context provided to the LLM. This prevents the LLM from hallucinating based on irrelevant or contextually inappropriate information while maintaining comprehensive coverage of relevant topics.
Solution Approach 2:
The patent converts the potential harm of retrieving many documents (including irrelevant ones) into a benefit by implementing a verification step that identifies and selects only those documents that genuinely contain answers. This transforms the overwhelming quantity of retrieved documents into a curated set of high-quality contextual information.
4Measurement precision
If strict semantic matching is used for document retrieval, then precision is improved, but recall deteriorates missing contextually relevant but superficially dissimilar information
Solution Approach 1:
The patent applies partial action by retrieving more documents than strictly necessary based on semantic similarity alone, then using an intermediary verification step to filter down to only those documents that actually contain answers. This allows the system to cast a wider net initially to capture contextually relevant but superficially dissimilar information, then refine the selection for precision.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are systems and methods for graph-based dynamic information retrieval and synthesis, particularly for enhancing the response generation of chatbot applications using large language models (LLMs). The methods involve receiving a user query, generating a query embedding, and accessing a knowledge graph comprising nodes representing documents with associated document embeddings. A set of documents is retrieved based on similarity scores, and a distance matrix is constructed to calculate shortest-path distances between nodes. Documents are clustered to identify a largest cluster from which content is retrieved to formulate a response. The disclosed embodiments improve upon retrieval augmented generation (RAG) by leveraging the hierarchical relationships of a knowledge graph and node interconnectedness to provide contextually coherent information and mitigate the inclusion of information from irrelevant or superficially related documents in LLM responses.