Hyperspace Sparse Partitioning for Scalable Vector Similarity Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector similarity search (VSS) methods face challenges with high computational complexity and resource-intensive efforts due to the curse of dimensionality and inefficient indexing strategies, particularly when dealing with large-scale vector storage.
Innovation Solution
A framework that partitions n-dimensional hyperspace into grid bins, maps vectors to these bins using their coordinates, and employs B-Trees for efficient lookup, avoiding empty bins and optimizing storage through sparse implementation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional vector similarity search methods are used, then search functionality is provided, but computational complexity increases and resource consumption increases due to the curse of dimensionality
Solution Approach 1:
The patent segments the n-dimensional hyperspace into multiple grid bins, dividing the search space into manageable regions. Each vector is mapped to specific grid bins based on its coordinates, allowing the search to be performed in a distributed manner across multiple bins rather than computing distances to all vectors, thus reducing computational complexity while maintaining search functionality
Solution Approach 2:
The patent introduces grid bins as intermediary structures between vectors and the search query. Instead of directly comparing the query vector with all stored vectors, the system uses grid bins to pre-organize and filter vectors, serving as an intermediate indexing layer that reduces the number of distance computations required
2Reliability
If dense hyperspace partitioning is used, then complete coverage of vector space is achieved, but storage requirements increase due to empty bins
Solution Approach 1:
The patent applies local quality by making the hyperspace partitioning sparse rather than dense. Instead of creating and maintaining all possible grid bins across the entire n-dimensional space, the system only creates and maintains grid bins that actually contain vectors. This means different regions of the hyperspace have different levels of granularity and presence, optimizing storage by eliminating empty bins while maintaining reliable coverage where vectors actually exist
Data Source
AI summary
A method for performing vector similarity search (VSS) includes: performing, based on a set of thresholds, a sparse partitioning of n-dimensional hyperspace into a plurality of grid bins to generate a hyperspace partition map, in which (i) only grid bins to which vectors mapped to are managed by the hyperspace partition map and (ii) each grid bin of the plurality of grid bins points to a storage location in a storage; ingesting a vector; using the hyperspace partition map to identify a grid bin of the plurality of grid bins for the vector and to identify coordinates of the grid bin; mapping the vector to the grid bin; and hashing the coordinates to generate a storage location key associated with a second storage location, in which the vector is stored to the second storage location using the storage location key.