RAG Document Ranking for Reliable Industrial AI Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current generative AI systems, particularly in industrial applications, produce unreliable outputs when prompted to generate content diverging from training data, lack transparency, and struggle with integrating disorganized documents, necessitating laborious manual filtering and domain expertise.
Innovation Solution
A computer-implemented method for a retrieval-augmented generation (RAG) system that includes an evaluation module to determine the importance of individual documents based on their contribution to output quality, optimizing the retrieval and augmentation modules to improve output reliability and transparency, using Shapley values and quality metrics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual filtering and organizing of documents is performed to ensure high-quality document input, then output reliability is improved, but labor effort and time consumption increase significantly
Solution Approach 1:
The system automatically evaluates and ranks documents using the evaluation module with quality metrics Q(Di) and importance values φi, eliminating the need for manual filtering. The RAG system self-optimizes by selecting high-quality documents autonomously based on computed importance values, making the system self-sufficient in document curation.
Solution Approach 2:
The patent introduces quantitative parameters (quality metrics Q, importance values φi, partial quality values Qp) to objectively evaluate and compare documents. By changing from subjective manual assessment to objective parameter-based evaluation, the system automatically identifies and selects high-quality documents without manual intervention.
2Reliability
If domain expertise is applied to manually organize documents for RAG system integration, then output quality is improved, but system complexity and resource requirements increase
Solution Approach 1:
The evaluation module automatically computes quality metrics and importance values for documents, enabling the system to self-organize and optimize document selection without requiring domain expert intervention. The system independently identifies high-quality documents through computational evaluation.
Solution Approach 2:
The patent replaces manual domain expert analysis with automated computational evaluation using quality metrics Q(Di) and importance values φi. This substitution of mechanical human analysis with automated algorithms reduces system complexity and resource requirements while maintaining or improving output quality.
3Loss of information
If all retrieved documents are used in the prompt, then information completeness is improved, but output reliability deteriorates due to inclusion of flawed or outdated documents
Solution Approach 1:
The patent applies different quality assessments to different documents individually. Each document Di is evaluated with its own quality metric Q(Di) and importance value φi, allowing the system to selectively include only high-quality documents in the prompt while excluding flawed or outdated ones, thus achieving local optimization of document quality.
Solution Approach 2:
Instead of using all retrieved documents, the system selects an optimal subset of documents with the highest partial quality value Qp. This partial action approach uses only the necessary high-quality documents required for reliable output, avoiding the harmful effects of including low-quality documents while maintaining information completeness.
4Adaptability or versatility
If RAG systems are applied in brownfield environments with disorganized documentation, then adaptability is improved, but manual preprocessing effort increases
Solution Approach 1:
The system automatically adapts to brownfield environments by autonomously evaluating and organizing disorganized documents using quality metrics and importance values. The RAG system self-preprocesses the documentation without requiring manual filtering or organizing, enabling easy integration into existing environments with unstructured documentation.
Data Source
Figure 1~2
Figure 3~4
AI summary
Providing reliability optimized output of a retrieval-augmented generation system Computer-implemented method for providing reliability-optimized output of a retrieval-augmented generation (RAG) system in an industrial application in reply to an input query (q), the RAG system (20) comprising a retrieval module (21), an augmentation module (22) and a generation module (23), comprising the steps: a) receiving (S1), by an evaluation module (24), a set (D) of documents determined by the retrieval module (21) depending on the input query (q), and an output (OPp) of the generator module (23) generated for a prompt (PRMT), wherein the prompt (PRMT) is generated by the augmentation module (22) by combining the input query (q) and the set (D) of documents, b) determining (S2), by the evaluation module (24), an importance value ϕi for each individual document (Di) in the set (D) of documents quantifying an importance of the individual document in relation to a quality value Q of the output (OPp) of the generator module (23) for the set (D) of documents, c) optimizing (S3) the retrieval module (21) and the augmentation module (22) with an ordered subset (S) of the set (D) of documents having a highest partial quality value (Qp) as observed during the determination of the importance value ϕi of each individual document (Di), and d) providing (S4) the output (QPf) of the generation module (23) predicted for an optimized prompt generated by the optimized augmentation module (22) combining the input query (q) with the subset (S) of documents with the highest partial quality value (Qp) for the input query (q) to a user interface.