Vector Retrieval Index Segmentation for Filtered Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vector retrieval methods in VectorDB require full-table scans, leading to inefficient retrieval processes due to the high time consumption of filtering records, which negatively impacts overall efficiency.
Innovation Solution
The method involves pre-generating candidate vector indexes based on candidate field values, allowing for targeted vector index queries by matching query vectors with filter conditions, thereby reducing the need for full-table scans and enhancing retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If full-table scan is performed for vector retrieval, then completeness of retrieval is ensured, but retrieval time consumption increases significantly
Solution Approach 1:
The patent segments the vector database into multiple candidate vector indexes based on different field values (e.g., category, tag). Instead of scanning the entire table, the system selects and scans only the relevant segment (vector index) that matches the filter conditions, significantly reducing retrieval time while maintaining completeness within the filtered results.
Solution Approach 2:
The patent pre-generates and pre-organizes candidate vector indexes before actual retrieval operations. These pre-built indexes are structured in advance based on potential filter conditions, allowing the retrieval system to quickly locate and access only the necessary data segments without performing full-table scans during actual queries.
2Productivity
If candidate vector indexes are pre-generated and selected based on filter conditions, then retrieval efficiency is improved, but system complexity increases
Solution Approach 1:
The patent creates a universal indexing structure where candidate vector indexes are pre-generated for multiple possible field values and stored in an organized manner. This multi-functional index structure can handle various filter conditions uniformly, allowing the system to efficiently retrieve data based on different criteria without requiring separate complex indexing mechanisms for each condition type.
Data Source
AI summary
The present disclosure provides a vector retrieval method and apparatus, a device and a medium, and relates to the field of artificial intelligence technologies, particularly the fields of cloud computing technologies, big data technologies, or the like. The vector retrieval method includes: acquiring a plurality of candidate vector indexes generated in advance, the plurality of candidate vector indexes being generated based on a plurality of candidate field values included in a target field of original data; acquiring a query vector and a filter condition, the filter condition being used for indicating a condition required to be satisfied by a target vector corresponding to the query vector; if the filter condition includes a target field value required to be satisfied by the target field, determining a target vector index corresponding to the target field value in the plurality of candidate vector indexes; and performing query in the target vector index based on the query vector to obtain the target vector.