Similarity Index Tree for Massive High-Dimension Dataset Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods are inefficient in handling massive, high-dimensional datasets for similarity-based queries, self-joins, and joins due to high computational complexity, particularly with the need to avoid the n^2 pairwise similarity computation phase.
Innovation Solution
The development of a method and apparatus for building a bulk similarity index tree, enabling fast similarity-based queries, self-joins, and joins through efficient hierarchical grouping and the use of anchor items to represent clusters, reducing computational complexity to O(n*log(n)) and O(n1*log(n1) + n2*log(n2)) respectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pairwise similarity computation is performed for all data members, then similarity accuracy is improved, but computational complexity increases to O(n^2)
Solution Approach 1:
The patent segments the dataset into clusters using hierarchical grouping, where each cluster is represented by an anchor item. Instead of computing pairwise similarity between all n items, the system divides items into groups and only computes similarity between cluster representatives (anchors), reducing the computational scope from O(n^2) to O(n*log(n)) for query operations and O(n1*log(n1) + n2*log(n2)) for join operations.
Solution Approach 2:
The patent introduces anchor items as intermediaries between query items and the dataset. Each cluster is represented by an anchor item that serves as a mediator: similarity computation is performed between the query item and anchor items first, then results are propagated to member items within clusters. This intermediary approach avoids direct pairwise comparison between all items while maintaining similarity accuracy through the transitive property within clusters.
2Speed
If hierarchical grouping with anchor items is used, then query speed is improved, but index building complexity increases
Solution Approach 1:
The patent performs preliminary action by pre-building a hierarchical index structure with anchor items before query execution. The index construction phase groups items into clusters and selects representative anchor items in advance. This preliminary organization enables fast query execution by avoiding runtime clustering computations, trading off one-time index building complexity for sustained query speed improvements.
3Measurement precision
If all pairwise similarities are computed, then join accuracy between datasets is improved, but processing time increases significantly
Solution Approach 1:
The patent extracts and utilizes the transitive property of similarity within clusters to avoid redundant computations. By computing similarity between cluster representatives (anchors) and propagating results to member items, the system extracts the essential similarity information without performing all pairwise comparisons. This extraction approach maintains join accuracy while dramatically reducing processing time from O(n1*n2) to O(n1*log(n1) + n2*log(n2)).
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and apparatus for fast similarity-based query, self-join, and join for massive high-dimension datasets utilizing a similarity index (400).