Vector Index Covering Columns for Predicate-Filtered Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector processing workloads face challenges in providing fast response times and efficient resource utilization for vector similarity searches, particularly in large databases with billions of vectors, due to the need for CPU-intensive indexing and searching operations, and the inefficiency of current vector indexes in handling predicates and non-vector data.
Innovation Solution
A vector database management system (VDBMS) that utilizes vector indexes with covering columns to accelerate operations by avoiding base table joins and reducing unnecessary vector distance calculations, employing techniques like Hierarchical Navigable Small Worlds Index (HNSW) and Inverted File Index (IVF) combined with predicate filtering to optimize speed and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional vector indexes are used for similarity search, then search speed is improved, but the ability to handle predicates and non-vector data is worsened
Solution Approach 1:
The patent merges vector index structures with traditional relational database index structures to create a unified index system that can handle both vector similarity search and predicate filtering. This combination allows the system to maintain fast search performance while gaining the ability to efficiently process predicates and non-vector data without requiring separate indexing mechanisms.
Solution Approach 2:
The patent creates a universal index structure that serves multiple functions: it performs vector similarity search, filters by predicates, and queries non-vector data all through a single index mechanism. This multi-functional approach eliminates the need for separate indexing systems and enables the index to adapt to various query types while maintaining performance.
2Reliability
If base table joins are performed during vector search, then data completeness is improved, but computing resources and response time are worsened
Solution Approach 1:
The patent merges the base table data with the vector index by storing covering columns (non-vector data) directly within the vector index structure. This integration allows the index to contain both vector data and associated non-vector data, eliminating the need for separate base table joins while maintaining data completeness.
Solution Approach 2:
The patent extracts necessary non-vector data (covering columns) from the base table and incorporates them into the vector index structure. By taking out only the required data and storing it within the index, the system reduces the amount of data that needs to be joined during queries, thereby reducing computing resources and response time.
3Measurement precision
If vector distance calculations are performed for all vectors, then search accuracy is improved, but CPU usage and processing time are worsened
Solution Approach 1:
The patent performs preliminary filtering by evaluating predicates and checking covering column values before computing vector distances. This preliminary action filters out vectors that cannot possibly satisfy the query conditions, reducing the number of expensive vector distance calculations that need to be performed while maintaining search accuracy.
Solution Approach 2:
The patent applies different quality levels of filtering at different stages of the search process. By using covering columns for quick filtering and only performing full vector distance calculations for vectors that pass the preliminary filter, the system optimizes the balance between search accuracy and CPU usage by applying appropriate quality of verification locally where needed.
4Reliability
If multiple joins between vector index and base table are performed, then query result completeness is improved, but device complexity and processing steps are worsened
Solution Approach 1:
The patent merges multiple processing steps into a single unified operation by integrating covering columns within the vector index structure. This allows the system to perform filtering, grouping, and data retrieval in one operation rather than requiring multiple separate joins between the vector index and base table, thereby reducing device complexity while maintaining result completeness.
Solution Approach 2:
The patent creates a universal index structure that can handle multiple query operations (similarity search, filtering, grouping) through a single processing path. This multi-functional approach eliminates the need for multiple specialized joins and processing steps, simplifying the overall device complexity while ensuring complete and accurate query results.
Data Source
AI summary
Techniques for efficiently processing vector queries against vector indexes are provided. In one technique, a vector query that includes a query vector and a predicate on a non-vector column of a base table that stores a plurality of vectors is received. In response and while traversing a vector index that is based on the plurality of vectors, a set of candidate vectors is identified based on the query vector. For each candidate vector in the set of candidate vectors, a value associated with that candidate vector and the non-vector column is identified in the vector index, and it is determined whether the value satisfies the predicate. A strict subset, of the set of candidate vectors, is selected whose values for the non-vector column satisfy the predicate. For each candidate vector in the strict subset, a vector distance between the query vector and the candidate vector is computed.