RAG Knowledge Base Relevance Scoring for Disjointed Enterprise Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data sources within organizations are disjointed, ephemeral, and contain extraneous data, making it difficult to generate high-quality summaries for user queries regarding products and services.
Innovation Solution
An active learning mechanism using a RAG chatbot that generates summaries from unstructured data, evaluates sentiment, removes personally identifiable information, and adjusts relevance scores based on user feedback to improve response quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RAG retrieves embeddings from authoritative data source, then context-relevance of output is improved, but retrieval accuracy deteriorates due to disjointed and ephemeral data sources
Solution Approach 1:
The system generates embeddings for all documents in the data source in advance and stores them in the knowledge base before queries are submitted. This preliminary action allows the RAG system to quickly retrieve pre-computed embeddings without dealing with the ephemeral nature of data sources at query time, thereby maintaining both retrieval accuracy and context-relevance
Solution Approach 2:
The patent introduces a knowledge base as an intermediary layer between the authoritative data source and the RAG system. This knowledge base stores pre-processed embeddings and acts as a stable intermediary that decouples the ephemeral data source from the retrieval process, ensuring consistent and accurate retrieval while maintaining context-relevance
2Reliability
If foundation model knowledge is updated frequently, then accuracy on current data is improved, but computational cost and time increase significantly
Solution Approach 1:
The system segments the knowledge update process into two independent parts: (1) updating the knowledge base with new documents and generating their embeddings, and (2) keeping the foundation model weights fixed. This segmentation allows frequent updates of the knowledge base without the computational burden of retraining the entire foundation model, thus maintaining accuracy while reducing update time
Solution Approach 2:
Instead of updating the foundation model directly with new data, the system creates copies of relevant information in the form of embeddings in the knowledge base. This copying approach allows the foundation model to remain unchanged while still providing accurate responses on current data through RAG, significantly reducing the time and computational cost of updates
Data Source
AI summary
A prioritization system receives documents from unstructured data sources across an organization and sanitizes the documents by summarizing the entries therein and removing personally identifiable information from the summaries. Additionally, the prioritization system determines relevance scores of each summary to related products/services and topics of frequently asked questions for the products/services. The summaries are stored in a knowledge base in association with their relevance scores. A chatbot engages in an active learning feedback loop with users by retrieving relevant summaries from the knowledge base according to the relevance scores when responding to user queries and increasing or decreasing relevance scores for summaries used in the responses based on positive or negative user feedback, respectively.