Vector Indexing via Locality-Sensitive Hashing for High-Dimensional Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database management systems face inefficiencies in performing high-speed vector neighbor searches, particularly for high-dimensional vectors, due to the lack of effective indexing methods for similarity degree calculations, leading to prolonged execution times and increased computational burdens.
Innovation Solution
A data management system employing locality-sensitive hashing (LSH) to preprocess feature vectors, generating peripheral vectors with reduced dimensions and building index information for efficient identification, allowing for rapid vector similarity searches by mapping vectors into a contracted space with discrete values, thereby reducing computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional database management systems perform vector neighbor searches for similarity calculations, then data retrieval functionality is provided, but the amount of calculations increases and execution time is prolonged
Solution Approach 1:
The patent applies preliminary action by pre-processing feature vectors into contracted vectors with discrete values before storage. This preprocessing step creates a contracted space representation that enables faster subsequent searches. The index building unit generates contracted vectors in advance and stores them with their corresponding original vectors, so that during search operations, the system can quickly compare contracted vectors without performing full high-dimensional vector calculations, thus reducing execution time while maintaining retrieval accuracy
Solution Approach 2:
The patent applies parameter changes by transforming the dimensionality and representation parameters of vectors. High-dimensional continuous feature vectors are converted into low-dimensional discrete contracted vectors through a contraction function. This parameter transformation reduces the computational complexity from O(N*D) to O(N) where N is the number of vectors and D is the dimensionality, enabling faster similarity searches while preserving the essential similarity relationships through the designed contraction function
2Loss of information
If high-dimensional feature vectors are stored and searched, then comprehensive data representation is achieved, but computational complexity increases
Solution Approach 1:
The patent applies the extraction principle by separating the feature vector representation into two parts: the original high-dimensional vectors are stored for complete information retention, while extracted contracted vectors with discrete values are used for computational operations. The extraction process identifies and preserves the essential similarity characteristics in the contracted form, allowing the system to perform calculations on the simplified representation while keeping the full-information original vectors available when needed
Solution Approach 2:
The patent transforms the parameter structure of vectors by changing from high-dimensional continuous parameters to low-dimensional discrete parameters through contraction. This parameter change reduces computational complexity from O(D) to O(1) per comparison while the carefully designed contraction function ensures that the essential information about vector similarity is preserved in the contracted representation
Data Source
AI summary
According to an embodiment, a data management system includes an index building unit and a searching unit. The index building unit generates a peripheral vector similar to a case example vector representing a feature vector of data to be stored, and builds index information of enabling identification of the case example vector corresponding to the generated peripheral vector. The searching unit refers to the index information in response to a search request in which a query vector representing an arbitrary feature vector is specified, identifies the case example vector corresponding to the peripheral vector that exactly matches with the query vector, and outputs a search result based on the identified case example vector.