Sharded Vector Database Querying for Non-Collocated Joins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle with efficiently processing vector queries across sharded databases due to the inefficiencies of non-collocated joins, which consume significant processing power and network bandwidth.
Innovation Solution
Implementing a vector query processor that determines the type of join (co-located or non-collocated) and employs pre-filter or post-filter approaches to optimize query processing, reducing unnecessary calculations and network traffic by managing joins and top-K results across multiple shards.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If non-collocated joins are used to process vector queries across sharded databases, then query processing can be performed, but processing power and network bandwidth consumption increase significantly
Solution Approach 1:
The patent segments the join operation into two phases: a filtering phase where each shard independently filters vectors based on pre-computed metrics, and a joining phase where only filtered results are combined. This segmentation reduces the amount of data that needs to be transferred and processed across shards, thereby reducing overall processing power consumption while maintaining query processing capability.
Solution Approach 2:
The patent performs preliminary filtering actions at each shard before the join operation. By pre-computing filter metrics and applying them locally, the system reduces the volume of data that needs to be transferred and processed during the actual join operation, thus reducing processing power and network bandwidth consumption.
2Productivity
If non-collocated joins are used to process vector queries across sharded databases, then query processing can be performed, but network bandwidth consumption increases significantly
Solution Approach 1:
The patent segments the data flow into local filtering operations at each shard and a subsequent joining operation. By performing filtering locally and only transferring filtered results between shards, the system significantly reduces network bandwidth consumption compared to transferring all data for processing.
Solution Approach 2:
The patent performs preliminary filtering actions at each shard before data needs to be transferred. This preliminary action reduces the volume of data that must be transmitted over the network, thereby reducing network bandwidth consumption while maintaining query processing capability.
3Measurement precision
If vector distance calculations are performed across all shards for non-collocated joins, then complete query results are obtained, but computational load increases
Solution Approach 1:
The patent segments the computational task by performing vector distance calculations only for filtered results at each shard rather than for all data. This segmentation maintains measurement precision for relevant results while significantly reducing the overall computational load across the system.
Solution Approach 2:
The patent performs preliminary filtering actions that reduce the dataset size before vector distance calculations are performed. By filtering data locally at each shard before computing distances, the system maintains accurate query results for relevant data while reducing the computational load required for distance calculations.
Data Source
AI summary
Techniques for querying sharded vector databases are provided. In one technique, a vector query that targets first and second sharded tables is received. In response, it is determined that the vector query includes a non-collocated join condition on the first and second sharded tables. In response, multiple shards storing the first and second sharded tables are identified, first data pertaining to the first sharded tables and second data pertaining to the second sharded table are retrieved from the shards, and a join operation is performed on the first data and the second data. A portion of results from the join operation is transmitted to each of the shards. Then, a top K result is received from each shard. A final top K result is generated based on the top K results from the shards. A response to the vector query is generated based on the final top K result.