Self-Supervised Retrieval Model for RAG Context Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Retrieval-Augmented Generation (RAG) and Dense Passage Retrieval (DPR) models face challenges such as the need for large annotated datasets, incorrect context retrieval leading to inaccuracies, difficulties in handling longer contexts and ambiguous queries, and slow retrieval speeds.
Innovation Solution
A self-supervised learning framework using contrastive learning for contextual retrieval augmented generation, which pre-trains a retrieval model on unlabelled texts by generating positive and negative query-context pairs through data augmentation, allowing for efficient and accurate context retrieval without relying on extensive human annotation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DPR models use encoding mechanisms to retrieve relevant documents, then retrieval accuracy is improved, but retrieval speed deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing dense vector representations of all documents in the corpus before retrieval operations. During retrieval, the query is encoded and compared against pre-computed document vectors using efficient similarity search algorithms, eliminating the need for real-time document encoding and improving retrieval speed while maintaining accuracy
2Measurement precision
If RAG models retrieve relevant documents to inform generation, then response accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary retrieval of relevant documents using dense passage retrieval before the generative process begins. By pre-fetching and ranking relevant passages based on query similarity, the system reduces the time required during generation while ensuring accurate context is available for the generative model
Solution Approach 2:
The patent extracts only the most relevant portions of documents that are necessary for answering the query, rather than processing entire documents. This selective extraction of pertinent information reduces processing time while maintaining response accuracy by focusing computational resources on relevant content
3Quantity of substance
If DPR models process longer documents, then comprehensive information retrieval is improved, but encoding effectiveness deteriorates
Solution Approach 1:
The patent segments long documents into smaller passages or chunks before creating dense vector representations. Each passage is encoded independently, allowing the model to capture local context effectively. The retrieval system then searches across these segmented passages, maintaining encoding effectiveness while achieving comprehensive information coverage through the collection of segments
Data Source
AI summary
The present disclosure leverages self-supervised learning to generate positive and negative question-context pairs, enabling the model to learn robust representations. This process involves data augmentation techniques to create variations of the original questions and contexts while preserving semantic relevance. A large corpus of unlabelled text data containing questions and their corresponding contexts, ensuring diversity and representativeness across various topics are used to train a self-supervised Large Language retrieval model.