Dual-Stage Vector Search with Quantized Filtering for RAG
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vector databases in AI applications face significant storage space bloat due to large embedding vectors and indexing structures, which worsens with increasing dataset sizes, impacting efficiency and accuracy in Retrieval-Augmented-Generation (RAG) operations.
Innovation Solution
A dual-stage vector search process involving quantization of embedding vectors to reduce their size, followed by re-ranking to restore accuracy, utilizing GPUs for computation, and employing lossless compression techniques to minimize storage and enhance search speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large embedding vectors (e.g., 1024 dimensions) are used to achieve good search accuracy, then measurement precision is improved, but storage space requirements increase significantly
Solution Approach 1:
The patent segments the vector search process into two distinct stages: (1) a coarse filtering stage using quantized vectors with reduced dimensions to eliminate irrelevant candidates, and (2) a precise ranking stage using full-precision vectors to accurately rank the filtered candidates. This segmentation allows the system to use small quantized vectors for most of the search space while reserving full-precision vectors only for the final candidate set, thereby reducing overall storage requirements while maintaining search accuracy.
Solution Approach 2:
The patent applies parameter changes by transforming the vector representation from full-precision floating-point format to quantized format with reduced bit depth. Specifically, it uses techniques such as binary quantization (reducing from 32-bit floats to 1-bit integers) or low-bit quantization (reducing to 8-bit or 16-bit integers). This parameter change dramatically reduces storage space while the dual-stage search process compensates for the loss in precision through the subsequent re-ranking using full-precision vectors.
2Measurement precision
If embedding vector dimensionality is increased to improve retrieval quality, then measurement precision is improved, but device complexity increases due to larger indexing structures
Solution Approach 1:
The patent segments the indexing structure into two parts: a quantized vector index for fast coarse filtering and a full-precision vector index for accurate ranking. The quantized index uses simplified data structures that require less memory and computation, while the full-precision index is only applied to a small subset of candidates. This segmentation reduces the overall complexity of the indexing system while maintaining retrieval quality.
3Quantity of substance
If quantized vectors are used to reduce storage space, then quantity of substance is reduced, but loss of information occurs reducing search accuracy
Solution Approach 1:
The patent applies preliminary action by first performing a coarse filtering search using quantized vectors to identify a candidate set of relevant items. Then, it performs a secondary ranking action using full-precision vectors on only this candidate set. This two-stage approach recovers the information lost during quantization for the final ranking, ensuring search accuracy is maintained while still benefiting from the storage efficiency of quantized vectors during the filtering stage.
Solution Approach 2:
The patent introduces an intermediary step between the quantized vector search and the final results. The intermediary is the re-ranking process that uses full-precision vectors to adjust and refine the results obtained from the quantized search. This intermediary step compensates for the information loss in quantization by re-evaluating the candidate results with higher precision, thereby restoring search accuracy.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
The disclosure describes system, devices, and methods for dual-stage vector search. In an example implementation, a method for operating a computer-implemented service is provided. The method includes receiving a context request for content with which to augment a prompt, generating a base vector based on input data in the context request and quantizing the base vector to produce a quantized vector. The method also includes searching a vector database to identify content items based at least on the quantized vector and obtaining the content items and generating base vectors for the content items. The method further includes selecting a subset of the content items based on at least on the base vector generated for the input data and the base vectors for the content items.