Multilevel Clustered Vector Data Search Using Cosine Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multilevel data clustering techniques for vector-based data face challenges in efficiently matching new vectors with large datasets, particularly in biometric applications, where high-dimensional vectors require stable and fast querying to maintain accuracy and reduce query time, which current methods fail to achieve effectively.
Innovation Solution
The method employs a multilevel clustering approach using cosine similarity to assign vectors to clusters and sub-clusters based on signed axes, allowing for near-constant time matching by determining the closest top-level and sub-cluster to a new input vector, thereby streamlining the comparison process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional multilevel data clustering techniques are used for vector-based data, then data can be organized into clusters, but query time increases linearly with data size and accuracy deteriorates as datasets grow larger
Solution Approach 1:
The patent divides the high-dimensional vector space into multiple levels of clusters and sub-clusters. Each level partitions the space further, creating a hierarchical structure. This segmentation allows the system to first identify candidate matches at coarser levels and then refine matching at finer levels, reducing the number of vectors that require full comparison while maintaining accuracy.
Solution Approach 2:
The patent introduces a hierarchical dimension to the traditional flat clustering approach by creating multiple levels of clusters (top-level clusters containing sub-clusters). This additional organizational dimension allows efficient pruning of the search space at upper levels before performing detailed comparisons at lower levels, achieving near-constant query time regardless of total data size.
2Productivity
If high-dimensional vectors are used for biometric data, then more features can be represented, but comparison and matching become computationally expensive and slower
Solution Approach 1:
The patent performs preliminary organization of vectors into hierarchical clusters and sub-clusters during data ingestion. This pre-computation creates an efficient search structure that enables rapid querying without requiring complex real-time computations. When a query vector arrives, the system leverages the pre-established hierarchical structure to quickly identify candidate matches.
Solution Approach 2:
The patent performs comparisons at multiple levels of granularity, starting with coarse top-level clusters and progressively refining to sub-clusters. This partial comparison approach avoids the excessive computational cost of comparing the query vector against all vectors in the database, achieving faster matching by performing only necessary comparisons at appropriate levels.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach stabilizes clustering, reduces query time, and maintains accuracy even as data grows, outperforming existing methods by enabling efficient retrieval of matching vectors in constant time, significantly improving performance in biometric data matching systems.
Implementation Method 1
Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space which measures the cosine of the angle between them.
Data Source
AI summary
A multilevel clustered data set for multidimensional vectors is created by defining a plurality of clusters based on each of the signed dimensions of the vectors, each dimension functioning as an axis. Vectors are assigned to each cluster by measuring cosine similarity between a vector and each axis. Sub-clusters are defined as ranges of cosine similarity values within a cluster, and each vector is assigned into the appropriate range based on their cosine similarity value with the axis of the cluster. Searching for a matching vector to a new vector is efficiently achieved in near-constant time by measuring cosine similarity for the new vector with each axis to identify the closest cluster, reusing the cosine similarity of the new vector and axis to determine which sub-cluster corresponds to the appropriate range of values, and then comparing each vector within the sub-cluster until a match is found or ruled out.