Similarity Search Indexing for Vector Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for finding similar pairs of objects in a dataset are inefficient, as they often require computing exact similarity scores for all pairs of vectors, leading to high computational complexity and storage requirements.
Innovation Solution
The approach involves identifying a comparison vector, estimating its similarity contribution, generating an index based on this contribution, and using it to efficiently identify similar vectors by applying a similarity threshold, thereby reducing the need for exact similarity score computations and optimizing storage and computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact similarity scores are computed for all pairs of vectors, then search accuracy is improved, but computational complexity and storage requirements increase significantly
Solution Approach 1:
The patent segments the similarity computation process into two stages: (1) computing exact similarity scores only for candidate pairs identified through indexing, and (2) filtering results using the index. This segmentation reduces the overall computational burden by avoiding exhaustive pairwise comparisons while maintaining search accuracy through the indexed candidate selection process.
Solution Approach 2:
The patent performs preliminary indexing of vectors before similarity computation. By pre-processing the data to create an index structure, the system can quickly identify candidate pairs that meet the similarity threshold without computing exact similarity scores for all pairs. This preliminary action reduces computational complexity while preserving accuracy for the final result set.
2Reliability
If all pairs of vectors are compared to find similar pairs, then completeness of search results is improved, but time consumption increases
Solution Approach 1:
The patent introduces an index structure as an intermediary between the vector database and the similarity computation process. The index acts as a mediator that pre-organizes vector information to enable efficient retrieval of candidate pairs, eliminating the need for exhaustive pairwise comparisons while ensuring complete search results through systematic candidate generation and filtering.
3Measurement precision
If similarity scores are computed for all vector pairs, then search thoroughness is improved, but storage requirements increase
Solution Approach 1:
The patent extracts and stores only the necessary indexing information and candidate pair identifiers rather than storing all exact similarity scores. By taking out only the essential data needed for efficient retrieval and comparison, the system maintains search thoroughness while significantly reducing storage requirements compared to storing complete similarity matrices for all vector pairs.
Data Source
AI summary
Pairs of similar objects in a population of objects can be found using a process that includes identifying a comparison vector x in a set of vectors having non-zero features, determining an estimated similarity contribution of a subset of features of the comparison vector x to a similarity between the comparison vector x and each vector in the set of vectors, generating an index that includes features based on a comparison of the similarity contribution with a similarity threshold, and identifying another vector in the set that is similar to the vector x using the index.