Knowledge Graph Re-Prompting for Consistent RAG Context Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Retrieval-Augmentation Generation (RAG) frameworks in Large Language Models (LLMs) face challenges in selecting the most relevant context from multiple matching document chunks, leading to inconsistent responses due to the lack of effective methods to clarify and ensure consistency in the context used for querying.
Innovation Solution
A knowledge graph is constructed to associate topics and keywords with document chunks, enabling the system to identify central ideas and pathways, and solicit follow-up prompts from users to clarify and ensure consistent context for accurate responses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple document chunks are retrieved to provide comprehensive context, then the completeness of information is improved, but the difficulty of selecting the most relevant context increases
Solution Approach 1:
A knowledge graph is introduced as an intermediary structure between the retrieved document chunks and the LLM. The knowledge graph organizes entities, relationships, and contextual information from multiple document chunks, enabling systematic analysis and selection of relevant context. This intermediary structure transforms the unstructured challenge of selecting relevant context into a structured process of graph traversal and entity relationship analysis.
Solution Approach 2:
The system implements feedback mechanisms where the LLM analyzes the knowledge graph structure and provides guidance on which contextual information is most relevant. The system iteratively refines context selection by evaluating the knowledge graph contents against the query requirements, allowing for adaptive selection of the most appropriate context from the retrieved document chunks.
2Measurement precision
If the system queries the LLM with multiple candidate chunks, then the accuracy of response can be improved, but the time required for processing increases
Solution Approach 1:
The knowledge graph is constructed and organized before the LLM queries are executed. Entities, relationships, and contextual information are pre-processed and structured in the knowledge graph, so that when the LLM needs to query about a topic, the relevant context is already organized and readily accessible, reducing the time required for processing while maintaining accuracy.
Solution Approach 2:
The context from multiple document chunks is segmented into discrete entities and relationships within the knowledge graph. This segmentation allows the system to query specific entities or relationship types rather than processing entire document chunks, significantly reducing processing time while maintaining the ability to provide accurate, targeted responses.
3Ease of operation
If the system uses simple similarity matching to select document chunks, then the ease of operation is improved, but the reliability of context selection deteriorates
Solution Approach 1:
The knowledge graph serves as an intermediary that enhances simple similarity matching by adding semantic understanding through entity relationships. While the initial retrieval can use efficient similarity matching for ease of operation, the knowledge graph structure provides additional reliability by organizing information semantically, allowing the system to verify and refine context selection based on entity relationships rather than relying solely on text similarity.
Data Source
AI summary
According to one aspect, a method includes obtaining, at an interface to a system that includes a large language model (LLM) arrangement, a first prompt, and identifying a plurality of candidate chunks of documents that substantially match the first prompt. The method also includes analyzing the plurality of candidate chunks to identify a chunk node associated with the plurality of candidate chunks, and generating a query arranged to solicit information associated with the plurality of candidate chunks. A second prompt is obtained in response to the query, and the plurality of candidate chunks is analyzed. Analyzing the plurality of candidate chunks using the second prompt includes identifying at least a first candidate chunk of the plurality of candidate chunks that is associated with the second prompt, wherein the first candidate chunk has a context. Finally, the method includes generating a response to the first prompt using the context.