Reference Database Gap Detection for Self-Updating RAG Knowledge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing chatbots and search engines using retrieval-augmented generation (RAG) frameworks rely on outdated information in reference databases, leading to inaccurate responses and inefficient computing resource consumption due to the delay in updating these databases, especially in customer support scenarios.
Innovation Solution
Implement a method for automatic detection and filling of information gaps in a reference database by monitoring support interactions, generating updated content using a large language model (LLM), and incorporating new information into the database in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual updating of reference database documentation is used, then accuracy of information can be maintained, but the process is slow and tedious causing delay in updating
Solution Approach 1:
The system enables automatic self-updating of the reference database by having the LLM monitor support interactions, detect information gaps, generate updated content, and update the database autonomously without manual intervention, transforming a manual process into an automated self-service system
Solution Approach 2:
The system implements feedback loops where the LLM continuously monitors support interactions, detects when information gaps occur, generates updates based on actual user queries and resolutions, and feeds this updated information back into the reference database, creating a continuous improvement cycle
2Reliability
If reference database is updated manually, then information accuracy can be maintained, but computing resource consumption increases due to repeated queries with outdated information
Solution Approach 1:
The system performs preliminary action by proactively monitoring support interactions and detecting information gaps before they affect multiple queries. By identifying and updating information gaps in advance, the system prevents repeated computing cycles of querying and re-querying with outdated information, thereby reducing overall computing resource consumption
3Reliability
If LLM retrieves data from reference database, then response generation quality improves, but the system cannot access real-time information from support interactions
Solution Approach 1:
The system establishes a feedback mechanism where support interactions are continuously monitored and the extracted information is fed back into the reference database. This ensures that the LLM always has access to the most current information from actual support scenarios when generating responses, eliminating the loss of real-time information
Solution Approach 2:
The system maintains continuous operation by permanently monitoring support interactions and continuously updating the reference database with new information. This continuous cycle ensures that the LLM always has access to up-to-date information without interruption, transforming a static reference database into a dynamically updated knowledge base
Data Source
AI summary
Methods and systems for automatic detection and filling of information gaps in a reference database are described. Responsive to a user query in an ongoing chat session, a query embedding associated with the user query is obtained. A synthetic question embedding is identified from a vector database, based on a similarity to the query embedding. Responsive to determining that the similarity between the synthetic question embedding and the query embedding does not meet a similarity threshold, the ongoing chat session is monitored to detect an answer to the user query. A prompt is provided to a large language model (LLM) to generate and display a textual content corresponding to the user query, based on the detected answer, for automatically updating the reference database. The disclosed methods and systems effectively incorporate new or undocumented information that is not currently captured within the reference database, as gaps are identified.