Random Draw Forest Index for Unstructured Data Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing locality sensitive hashing (LSH) based indexing and search algorithms for unstructured data are biased towards the most significant bits (MSB) of compact feature vectors, leading to inaccurate and inefficient similarity searches due to the prioritization of these bits in partitioning, which can result in incorrect nearest neighbors being returned.
Innovation Solution
The method involves generating a random draw forest (RDF) index structure by shuffling the hash values of compact feature vectors using random shuffling permutations, creating multiple index tables for each data object, and using these shuffled versions to index and search for similar data objects, thereby mitigating the bias towards any particular bit order and improving search accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing LSH-based indexing methods use the first several bits (MSB) of compact feature vectors to identify similar feature vectors, then the indexing process is simple and efficient, but the search accuracy deteriorates because these bits are a poor indicator of similarity
Solution Approach 1:
The patent segments the compact feature vector into multiple groups of bits, where each group is used to generate a separate hash value. Instead of relying solely on the MSB, the system divides the feature vector into multiple segments and processes each segment independently to create multiple hash values, thereby distributing the importance across different bit positions rather than concentrating it in the MSB.
Solution Approach 2:
The patent transforms the single-dimension MSB-based indexing into a multi-dimensional hashing approach by generating multiple hash values from different bit groups. This creates a multi-dimensional index structure where each dimension corresponds to a different bit group, allowing the system to explore similarity from multiple perspectives rather than being constrained to a single MSB-based dimension.
2Device complexity
If existing LSH-based indexing methods prioritize MSB in partitioning, then the index structure is simple to maintain, but the nearest neighbor identification becomes inaccurate
Solution Approach 1:
The patent segments the feature vector into multiple bit groups and generates multiple hash values, creating a segmented index structure. Each segment contributes to the overall indexing process, distributing the complexity across multiple simple hash computations rather than relying on a single complex MSB-based partitioning scheme.
Solution Approach 2:
The patent merges multiple hash values generated from different bit groups into a unified indexing process. The system combines the results from multiple independent hash computations to identify candidate neighbors, merging the strengths of different bit group comparisons to achieve more accurate and reliable nearest neighbor identification.
3Use of energy by moving object
If existing methods use consecutive bits for hashing, then the hashing process is computationally efficient, but the search results are biased towards MSB similarities
Solution Approach 1:
The patent segments the feature vector into non-consecutive or differently arranged bit groups for hashing. Instead of using consecutive bits in a single sequence, the system divides the bits into multiple segments that can be processed independently, allowing efficient computation while reducing the bias towards any particular bit position including MSB.
Solution Approach 2:
The patent changes the dimensionality of the hashing approach by generating multiple hash values from different bit groupings. This multi-dimensional hashing strategy maintains computational efficiency through parallel processing of independent bit groups while improving similarity measurement accuracy by considering multiple perspectives simultaneously.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
System and method of generating an index structure for indexing a plurality of unstructured data objects, including: generating a set of compact feature vectors, the set including a compact feature vector for each of the data objects, the compact feature vector for each data object including a sequence of hashed values that represent the data object; generating a plurality of twisted compact feature vector sets for each of set of compact feature vectors, each of the twisted compact feature vector sets being generated by applying a respective random shuffling permutation to the set of compact feature vectors; and for each twisted compact feature vector set, generating an index for the data objects in which the data objects are slotted based on sequences of hashed values in the twisted compact feature vector set.