Multi-Vector Search with a Covering Column Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-vector searches require excessive processing power and memory storage due to the complexity of maintaining and discarding top documents and their corresponding chunk vectors during the search process, leading to inefficient processor and memory usage.
Innovation Solution
Implementing a multi-vector search method that utilizes a chunk candidates heap, entity hash table, and entity results heap, along with a covering column index, to efficiently manage and organize search results, thereby optimizing processor and memory efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional multi-vector search methods are used to search for top documents and their chunk vectors, then search completeness is improved, but processor power consumption and memory usage increase excessively
Solution Approach 1:
The patent extracts and separates the document identification task from the chunk vector search task. By using a covering column index to store only document identifiers rather than full chunk vectors, the system reduces memory usage and processing requirements while maintaining the ability to return complete document results with their top matching chunks.
Solution Approach 2:
The patent creates a simplified copy of the search index using a covering column that stores only the essential document identifiers. This copy is used for the search process, allowing the system to work with much smaller data structures while still achieving complete search results through the relationship between the covering column and the full vector data.
2Measurement precision
If traditional multi-vector search methods maintain all chunk vectors during search, then search accuracy is improved, but memory storage requirements increase excessively
Solution Approach 1:
The patent extracts only the necessary document identifier information from the full chunk vectors and stores it in the covering column index. This extraction allows the system to maintain search accuracy by preserving the essential mapping between documents and their vectors while dramatically reducing memory storage requirements.
Solution Approach 2:
The patent segments the search data into two parts: the covering column containing lightweight document identifiers for search operations, and the full vector data stored separately for accurate matching. This segmentation allows the system to use minimal memory during search while maintaining accuracy through the separate full vector storage.
3Measurement precision
If detailed tracking of top documents and chunk vectors is performed, then search result quality is improved, but device complexity increases
Solution Approach 1:
The patent extracts the complex tracking of individual chunk vectors and replaces it with tracking only the document identifiers in the covering column. This simplification reduces device complexity by eliminating the need to manage and track numerous chunk vectors during the search process, while search result quality is maintained through the covering column's mapping to the full vector data.
Data Source
AI summary
A plurality of item-chunk queues, an item map, and an item results queue are maintained. A plurality of vectors are identified, each vector corresponding to a different chunk of a plurality of chunks. For each vector of the plurality of vectors: an item identifier of the each vector is identified; an item-chunk queue that corresponds to the item identifier is identified; and a determination is made as to whether the each vector is to be added to the item-chunk queue. A particular item identifier is stored in an entry, of the item map, that references a particular item-chunk queue of the plurality of item-chunk queues. A determination is made as to whether to add the at least one vector to the particular item-chunk queue. A determination is made as to whether to update the item results queue based on the at least one vector.