Two-Level Clustering Indexing for High-Dimensional Vector Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity of search objects and the exponential increase in search difficulty due to the need for high-dimensional vector comparisons in large datasets lead to inefficient similarity searches, making brute-force methods impractical for user requirements.
Innovation Solution
A method and device for pre-clustering samples at two levels and establishing indexes at two levels, allowing for efficient querying of similar samples through two-stage retrieval, where the first level reduces the sample range and the second level converts high-dimensional vector calculations into low-dimensional segment vector operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If brute-force search is used to perform similarity searches on complex objects represented by high-dimensional vectors, then comprehensive search coverage is achieved, but query time becomes excessively long and calculation amount becomes very large
Solution Approach 1:
The patent segments high-dimensional feature vectors into multiple low-dimensional segment vectors. Each segment vector is independently clustered and indexed, transforming a single complex high-dimensional search into multiple simpler low-dimensional searches. This segmentation reduces the computational complexity of each comparison while maintaining overall search effectiveness through aggregation of segment results.
Solution Approach 2:
The patent performs preliminary clustering and indexing of segment vectors before actual query execution. By pre-organizing data into cluster structures with stored cluster centers and segment assignments, the system prepares the data in advance to enable rapid query processing. This preliminary action eliminates the need for exhaustive pairwise comparisons during query time.
2Measurement precision
If brute-force search is used to compare distances between high-dimensional vectors, then accurate similarity measurement is achieved, but calculation amount becomes very large
Solution Approach 1:
The patent divides high-dimensional vectors into multiple low-dimensional segment vectors, reducing the dimensionality of each comparison operation. By computing similarities in segmented space rather than full high-dimensional space, the calculation amount per operation is dramatically reduced while the aggregation of multiple segment similarities maintains overall measurement accuracy.
Solution Approach 2:
The patent introduces cluster centers as intermediary representations. Instead of directly comparing every pair of high-dimensional vectors, the system uses pre-computed cluster centers as mediators to guide the search process. This intermediary approach reduces the number of direct comparisons needed while preserving similarity measurement accuracy through the hierarchical clustering structure.
3Speed
If two-level pre-clustering and indexing is implemented, then query speed is significantly accelerated, but system complexity increases
Solution Approach 1:
The patent segments both the feature vectors and the indexing process into multiple levels. The first level clusters complete feature vectors, while the second level clusters segment vectors within each first-level cluster. This segmented hierarchical indexing structure enables faster queries by progressively narrowing down candidate samples through multiple filtering stages, with each stage operating on reduced-dimensional data.
Solution Approach 2:
The patent adds a hierarchical dimension to the indexing structure by creating two levels of clustering. The first level provides coarse-grained grouping, and the second level provides fine-grained segmentation. This dimensional expansion in the indexing hierarchy allows the system to manage complexity through structured organization rather than monolithic processing, improving query speed through multi-stage filtering.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Implementations of the present specification provide classification and indexing methods and devices, and methods and devices for querying similar samples. During classification, the samples in the sample set are clustered at two levels, and the clustering results are recorded in a first vector table and a second vector table. During indexing, indexes are established at two levels for each sample in the sample set, where the first level index points to a coarse cluster center to which the sample belongs, and the second level index points to a segment cluster center corresponding to a segment vector of the sample. During query of similar samples, searches are performed at two levels on the query samples. The first-level search is to determine a coarse cluster center that is closer to the query sample from the first vector table obtained through classification, and obtain comparison samples that belong to the coarse cluster center. The second-level search is to select a comparison sample whose distance meets a predetermined criterion as a similar sample. As such, retrieval and query of samples are accelerated.