Vector Search System Partial Similarity Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pattern retrieval methods, such as those described in PTL1 and PTL2, face challenges in achieving high k-nearest neighbor accuracy rates while speeding up the search process, and are limited to using Euclidean distance as a measure of similarity, which is insufficient for applications requiring high accuracy like face or voice authentication.
Innovation Solution
A search system that calculates partial similarity between input and registered vectors for select dimensions, determines an upper limit of similarity, and decides whether to reject vectors based on this limit, allowing for faster search processing without degrading accuracy, and enabling the use of various similarity measures beyond Euclidean distance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If simple repetition of similarity calculation is performed for all registered patterns, then k-nearest neighbor accuracy rate is maintained, but computational cost increases significantly
Solution Approach 1:
The patent segments the similarity calculation process into two stages: first calculating similarity using only a subset of dimensions (partial similarity), then using this result to determine whether to perform full similarity calculation. This segmentation allows the system to quickly filter out obviously dissimilar patterns without computing full similarity for all registered patterns, thereby maintaining accuracy while improving search speed.
Solution Approach 2:
The patent applies partial action by calculating similarity for only some dimensions initially rather than all dimensions. The system computes partial similarity using a subset of feature dimensions, and only performs complete similarity calculation for patterns that pass the partial similarity threshold, thus avoiding excessive computation on clearly irrelevant patterns.
2Productivity
If approximate nearest neighbor search methods (LSH, Spectral Hashing) are used to speed up search, then search speed improves, but k-nearest neighbor accuracy rate decreases
Solution Approach 1:
The patent performs preliminary similarity calculation using a subset of dimensions before committing to full similarity computation. By预先 (in advance) evaluating partial similarity, the system can eliminate clearly dissimilar patterns early, avoiding the need for complex approximate methods like LSH or Spectral Hashing that inherently sacrifice accuracy for speed.
Solution Approach 2:
The patent introduces partial similarity calculation as an intermediary step between the input pattern and full similarity comparison. This intermediary evaluation using selected dimensions acts as a filter that preserves accuracy for promising candidates while discarding irrelevant ones, avoiding the accuracy loss associated with direct approximate nearest neighbor methods.
3Productivity
If cascading method with low-dimensional feature vectors is used, then search speed improves, but only Euclidean distance can be used as similarity measure
Solution Approach 1:
The patent segments the feature dimensions and selects only relevant subsets for partial similarity calculation, rather than using all dimensions in cascading stages. This segmentation allows the use of various similarity measures (cosine similarity, Manhattan distance, etc.) on the selected dimensions while maintaining search efficiency.
Solution Approach 2:
The patent applies local quality by selecting specific dimensions that are most relevant for the particular application or data characteristics. Different dimensions can be weighted or selected based on their importance, allowing the system to use appropriate similarity measures for different parts of the feature space rather than being constrained to a single Euclidean distance approach.
4Productivity
If indices are created for approximate nearest neighbor search, then search speed improves, but creation of indices increases device complexity
Solution Approach 1:
The patent extracts and uses only the necessary feature dimensions for partial similarity calculation, avoiding the need to create and maintain complex index structures. By taking out only the essential dimensions for quick filtering, the system achieves speed improvement without the overhead of LSH or Spectral Hashing index creation and maintenance.
Data Source
AI summary
Provided is a search system which is configured to search for a registered vector being similar to an input vector among a plurality of registered vectors, on the basis of a degree of similarity between an input vector and a registered vector. The search system includes a partial similarity calculation unit that calculates a degree of partial similarity which is the degree of similarity concerning some of one or more dimensions of the input vector and the registered vector, a limit calculation unit that calculates, on the basis of the degree of partial similarity, an upper limit of the degree of similarity that is expected when the degree of similarity is calculated, and a rejection decision unit that decides, on the basis of the upper limit of the degree of similarity, whether or not to reject the registered vector from a candidate for a search result.