Snapshot-Based Vector Index Updates for RAG Dataset Mutations

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing AI model training methods require re-indexing the entire input data set when changes occur, consuming significant network bandwidth and computational resources, especially in large-scale distributed storage systems.

Innovation Solution

Implementing a snapshot difference processor (snapdiff processor) to detect and update only modified documents or portions of documents in the input data set, using point-in-time snapshots to generate a difference report and update vector indexes accordingly, reducing the need to reload the entire data set.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the entire input data set is re-indexed when changes occur, then the AI model responses remain accurate and up-to-date, but network bandwidth and computational resources are consumed excessively

Engineering Contradiction:

Improveaccuracy of AI model responsesVSAvoidnetwork bandwidth and computational resources

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The patent segments the input data set into individual documents or portions, enabling selective indexing of only changed segments rather than re-indexing the entire data set. The snapshot difference processor compares snapshots to identify specific modified documents, extracts only those changed portions, and updates the vector index with selective portions, thereby reducing computational and network resources while maintaining response accuracy.

Inventive Principle:

Principle #1Segmentation

2Reliability

If the entire input data set is re-indexed when changes occur, then the vector index remains up-to-date, but the processing time and computational overhead increase significantly

Engineering Contradiction:

Improveup-to-date status of vector indexVSAvoidprocessing time for index updates

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent implements preliminary action by taking snapshots of the input data set at specific points in time before changes occur. These snapshots are stored and later compared to detect changes. By having these pre-captured snapshots ready, the system can quickly identify modified documents without performing full data set re-processing, thus reducing processing time while keeping the vector index current.

Inventive Principle:

Principle #10Preliminary action

3Loss of energy

If only modified documents are updated in the vector index, then network resources are conserved, but the complexity of detecting and managing changes increases

Engineering Contradiction:

Improvenetwork bandwidth consumptionVSAvoidcomplexity of change detection mechanism

Core Design Contradiction:

Loss of energyVSDevice complexity

Solution Approach 1:

The patent uses copying by creating snapshots - copies of the input data set at specific points in time. These snapshots are stored in storage and later compared to detect changes. Instead of directly monitoring and detecting changes in the live data set (which would be complex), the system copies the data at intervals and compares these copies, simplifying the change detection mechanism while still enabling selective updates.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20250342179A1Methods and apparatus to manage input data sets to reflect dataset mutations for genai and rag applications

Publication Date: 2025.11.06 CLOUDERA INC

AI summary

Disclosed examples include analyzing a difference report indicative of at least one change between a first snapshot of an input data set in a storage system at a first time and a second snapshot of the input data set in the storage system at a second time; updating a vector index based on a change indicator in the difference report; and sending a refresh notification to a large language model (LLM) query engine based on the update of the vector index.