Vector Database Retrieval Using Maximum Dispersion Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current vector databases lack the capability to efficiently retrieve diverse or dissimilar data from an input query, as they are primarily designed for similarity search rather than diversity search, which is necessary for applications like sampling differing opinions or identifying distinct types of data.
Innovation Solution
Implement a modified Gaussian Mixture Model (GMM) algorithm that utilizes vector database indexing with clustering to efficiently compute the furthest vector from a query vector, using centroids to reduce computational complexity, and iteratively add diverse vectors to a subset, ensuring maximum pairwise distance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional vector database similarity search methods are used, then retrieval speed is efficient, but the system cannot retrieve diverse or dissimilar data from the query
Solution Approach 1:
The patent segments the vector database into multiple clusters, where each cluster represents a group of semantically similar vectors. This segmentation allows the system to efficiently search for diverse vectors by selecting from different clusters rather than comparing against all vectors individually, thus maintaining retrieval efficiency while enabling diversity retrieval capability.
Solution Approach 2:
Instead of finding vectors similar to the query (traditional similarity search), the patent inverts the approach by finding vectors that are maximally dissimilar to the query. This is achieved by selecting clusters whose centroids are farthest from the query vector and then selecting vectors within those clusters that maximize the minimum distance to all previously selected vectors.
2Measurement precision
If exhaustive search methods are used to find maximally dispersed vectors, then retrieval accuracy is high, but computational complexity increases significantly
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing cluster centroids before the actual diversity retrieval query. This allows the system to quickly identify candidate clusters based on centroid distances without having to compute distances to all individual vectors, significantly reducing computational complexity while maintaining accuracy through the iterative selection process.
Solution Approach 2:
The patent introduces cluster centroids as intermediaries between the query vector and the actual vector selection process. Instead of directly comparing the query vector to all stored vectors, the system first compares the query to cluster centroids to identify candidate clusters, then performs refined selection within those clusters. This intermediary approach dramatically reduces the search space and computational complexity.
3Manufacturing precision
If all vectors are considered for diversity retrieval, then the most diverse subset is found, but the retrieval process becomes computationally infeasible for large databases
Solution Approach 1:
The patent segments the large vector database into multiple clusters, reducing the problem from searching through all vectors to searching within selected clusters. This segmentation maintains the ability to find optimal diverse subsets by ensuring that diverse vectors are likely to be distributed across different clusters, while dramatically improving retrieval speed by limiting the search to relevant clusters only.
Solution Approach 2:
The patent applies partial action by selecting only the necessary number of diverse vectors (k vectors) rather than processing all vectors in the database. The iterative algorithm stops once the desired subset size is reached, performing computations only on the minimal necessary data while still guaranteeing optimal diversity for the selected subset through the max-min distance optimization.
Data Source
AI summary
A query vector is directed to a vector database that stores a plurality of vectors that are indexed into a plurality of clusters. In response to receiving the query vector, a furthest vector is found with an approximate maximum distance from the query vector by selecting a furthest cluster having a centroid furthest from the query vector. The query vector and furthest vector are placed into a subset. At least one diverse vector is added into the subset by performing one or more repetitions involving: determining another furthest cluster having another centroid furthest from all vectors in P; selecting another vector from the other furthest cluster that is furthest from all the vectors in P; and inserting the other vector into P. The subset P is used to provide a response to a diversity query.