Cardinality-Adaptive Vector Search with HNSW Space Expansion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approximate nearest neighbor (ANN) search methods in vector databases suffer from performance degradation due to varying search performance based on data filtering conditions, particularly when high cardinality filtering reduces recall accuracy.
Innovation Solution
A method that statistically processes cardinality data and compares it with a predetermined threshold to determine optimal data filtering and search strategies, using KNN for high cardinality and expanding the search space with HNSW for low cardinality, optimizing the search algorithm to maintain accuracy and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If high cardinality filtering is applied during ANN search, then search speed is improved, but recall accuracy deteriorates
Solution Approach 1:
The patent applies dynamics by making the search algorithm adaptive based on cardinality conditions. When cardinality is below the threshold, the system dynamically switches from standard ANN search to a modified search that expands the search space by a multiple of the inverse cardinality value, thereby maintaining recall accuracy while still benefiting from filtering in high-cardinality scenarios.
Solution Approach 2:
The patent changes the search parameter (search space expansion multiple) based on the cardinality parameter. By calculating the inverse of the cardinality value and using it as a expansion multiple, the system adjusts the search behavior to compensate for the restrictive filtering effect, thus resolving the contradiction between speed and accuracy.
2Productivity
If general data filtering is performed followed by KNN search, then search performance is improved for high cardinality, but accuracy deteriorates for low cardinality
Solution Approach 1:
The patent segments the search process into two distinct paths based on cardinality threshold: one path for high cardinality data using general filtering followed by KNN search, and another path for low cardinality data using modified ANN search with expanded search space. This segmentation allows each path to be optimized for its specific cardinality range.
Solution Approach 2:
The patent applies local quality by using different search strategies tailored to local conditions (cardinality ranges). High cardinality regions use aggressive filtering for speed, while low cardinality regions use conservative search expansion for accuracy, making the overall system adaptable to local data characteristics.
Data Source
AI summary
Provided are a method and system for searching for a nearest neighbor in a cardinality-based vector database. Cardinality data of a filtering condition column is statistically processed and recorded, and a data filtering and search method is determined by comparing the recorded data with a predetermined threshold value. When the cardinality data is higher than the threshold value, general data filtering and k-nearest neighbor (KNN) search are performed. When the cardinality data is lower than the threshold value, a hierarchical navigable small world (HNSW) algorithm is used, and a search space is expanded by modifying a search algorithm based on an inverse value of the cardinality or a filter combination probability. In this case, the ef-search value is increased by a corresponding multiple to expand a search space in a greedy search process for a candidate set removed by filtering. The predetermined threshold value may be determined when the number of cardinality eigenvalues is 25 or more or when the filter combination probability is 4% or less.