Hierarchical Bin Search in Associative Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining algorithms face challenges in performing fast and accurate similarity searches in high-dimensional spaces, which are essential for identifying similar items in large datasets.
Innovation Solution
A similarity search system utilizing a hierarchical database of bins stored in an associative memory array, where each bin is identified by an order vector representing original vectors, and the system includes a similarity searcher that operates on multiple columns simultaneously to find similar bins to a query vector.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional similarity search algorithms are used in high-dimensional spaces, then search accuracy can be maintained, but search time and computational complexity increase significantly
Solution Approach 1:
The patent segments the high-dimensional search space by introducing a hierarchical database structure with multiple levels. The search process is divided into coarse-grained filtering at upper levels and fine-grained verification at lower levels, allowing the system to quickly eliminate large portions of the search space while maintaining accurate results through progressive refinement.
Solution Approach 2:
The patent transforms the high-dimensional similarity search problem into a lower-dimensional problem by projecting data onto a reduced-dimensional space using dimensionality reduction techniques. This allows the system to perform searches in a compressed space while preserving the essential similarity relationships, significantly reducing search time while maintaining accuracy.
2Productivity
If the dimensionality of the search space is reduced, then search speed improves, but storage utilization and search accuracy may deteriorate
Solution Approach 1:
The patent performs preliminary dimensionality reduction and data transformation during the database construction phase, organizing data into a hierarchical structure with pre-computed embeddings at multiple levels. This preliminary processing enables faster searches without sacrificing accuracy, as the reduced-dimensional representations are pre-optimized to preserve similarity relationships.
Solution Approach 2:
The patent employs multi-level dimensionality reduction where data is represented at different dimensionalities across hierarchical levels. Upper levels use lower-dimensional representations for fast filtering, while lower levels progressively restore detail, ensuring that search speed improves through dimensionality reduction while accuracy is maintained through progressive refinement.
3Productivity
If a hierarchical database structure is implemented, then search complexity reduces to O(1), but system complexity and memory requirements increase
Solution Approach 1:
The patent implements a nested hierarchical database structure where databases at different levels are organized in a tree-like manner, with each level containing references to the next. This nested structure allows the system to achieve O(1) search complexity by navigating through predefined hierarchical paths, while the regular structure of nested databases makes the increased system complexity manageable through standardized operations.
Data Source
AI summary
A similarity search system includes a database of original vectors, a hierarchical database of bins and a similarity searcher. The hierarchical database of bins is stored in an associative memory array, each bin identified by an order vector representing at least one original vector and the dimension of the order vector is smaller than the dimension of the original vector. The similarity searcher searches in the database for at least one similar bin whose order vector resembles an order vector representing a query vector and provides at least one original vector represented by the bin resembling the query vector.