Multi-Vector Retrieval Pipeline for Low-Memory Re-Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information retrieval systems face challenges with high memory requirements and latency due to the large index sizes of multi-vector information retrieval models, particularly in implementing scalable systems.
Innovation Solution
The system offloads re-ranking embeddings to storage drives, such as SSDs, reducing memory requirements and maintaining near-memory query latency through an approximate nearest neighbor-based prefetching mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-vector information retrieval models are implemented, then retrieval accuracy is improved, but memory requirements increase significantly
Solution Approach 1:
The patent divides the index into multiple segments stored in different memory hierarchies (fast memory and storage). Single-vector indices are separated from multi-vector indices, allowing the system to maintain accurate multi-vector representations while reducing the memory footprint available to the query processing path. This segmentation enables retrieval accuracy to be preserved through careful orchestration of segmented index components.
Solution Approach 2:
The patent extracts the computationally intensive multi-vector embedding calculations from the main query processing path and pre-computes them offline. The extracted multi-vector representations are stored in storage, while only lightweight single-vector indices remain in fast memory for query processing. This extraction eliminates the need to hold large multi-vector structures in memory during queries while preserving retrieval accuracy through the pre-computed representations.
2Measurement precision
If large index sizes are used for multi-vector models, then retrieval quality is improved, but system latency increases
Solution Approach 1:
The patent performs preliminary action by pre-computing multi-vector embeddings and storing them in advance. During query processing, the system uses pre-computed single-vector indices for rapid candidate generation, avoiding the need to perform expensive multi-vector calculations in real-time. This preliminary computation phase separates the quality-enhancing embedding generation from the time-critical query response path.
Solution Approach 2:
The patent segments the retrieval process into distinct phases: candidate generation using single-vector indices from fast memory, and re-ranking using pre-computed multi-vector representations from storage. This segmentation allows the time-critical path to use lightweight structures while the quality-enhancing but slower multi-vector operations are performed separately, preventing latency degradation.
3Measurement precision
If multi-vector representations are stored in memory, then retrieval accuracy is maintained, but device complexity increases
Solution Approach 1:
The patent introduces single-vector indices as intermediaries between the query and the multi-vector representations stored in storage. These intermediary structures enable the system to query the large multi-vector index without loading it into memory, by translating queries into the intermediary single-vector space that can be efficiently maintained in fast memory while preserving access to the comprehensive multi-vector representations.
4Productivity
If scalable information retrieval systems are built, then system capacity is improved, but memory requirements increase
Solution Approach 1:
The patent transitions from a single-memory-hierarchy architecture to a multi-dimensional memory architecture, utilizing both fast memory and storage layers. This dimensional change allows the system to scale capacity by leveraging storage for bulk data while maintaining fast access paths through selective use of single-vector indices in fast memory, breaking the direct correlation between capacity and fast memory requirements.
Data Source
AI summary
Provided are systems, methods, and apparatuses for systems and methods of memory efficient multi-vector information retrieval based on embeddings from a storage pipelined network. In one or more examples, the systems, devices, and methods include performing a first portion of a nearest neighbor search on a first subset of a set of media sources; performing a fetch process on the first subset that include identifying a first set of highest matching media sources from the first subset, transferring multi-vector representations of the first set of highest matching media sources from a storage drive to a memory, and performing a second ranking of the first set highest matching media sources. The systems, devices, and methods include performing, in parallel with the fetch process, a second portion of the nearest neighbor search on a second subset of the set of media sources.