LSH Feature Selection and Pruning for Skewed Similarity Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing locality sensitive hashing (LSH) methods face scalability issues due to data skewness, particularly in similarity search applications, as they are designed for exact similarity search which is inefficient and limited by skewed feature distributions.
Innovation Solution
Implement feature selection and pruning techniques using Inverse Leaf Frequency (ILF) and Normalized Sub-Path Frequency (NSPF) scoring to select relevant features, transform them into vectors, and apply random hash functions to reduce skewness, enabling efficient approximate similarity search.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact similarity search is used, then measurement precision is improved, but productivity deteriorates due to scalability issues
Solution Approach 1:
The patent extracts and removes the problematic skewed features from the dataset using feature selection and pruning techniques. By identifying and eliminating features that cause data skewness (such as features with extreme frequency values), the system retains only the most relevant features for similarity search, thereby maintaining accuracy while improving processing speed.
Solution Approach 2:
Instead of directly optimizing exact similarity search for skewed data, the patent inverts the approach by using feature selection to transform the skewed data into a more balanced representation. This inverted strategy of selecting a subset of features rather than processing all features enables the system to achieve both precision and productivity goals.
2Measurement precision
If all features are used in LSH, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent extracts only the most relevant features from the complete feature set using scoring mechanisms. By removing redundant and less important features, the system reduces the dimensionality of the data fed into the hash table, thereby decreasing device complexity while preserving the essential information needed for accurate similarity search.
Solution Approach 2:
The patent applies partial action by using a subset of features rather than all available features. This selective approach processes only the most informative features through the LSH mechanism, reducing the complexity of the hash table structure and operations while maintaining sufficient precision for effective similarity search.
3Productivity
If feature selection is applied, then productivity is improved, but measurement precision may deteriorate
Solution Approach 1:
The patent applies local quality by differentiating the treatment of individual features based on their specific characteristics. Rather than uniformly processing all features, the system assigns different weights and selection priorities to different features based on their relevance, frequency, and contribution to similarity measurement. This localized optimization ensures that the most important features are retained for precision while less critical features are pruned for productivity.
Data Source
AI summary
Systems and methods for counteracting data-skewness for locality sensitive hashing via feature selection and pruning are disclosed. In one embodiment, a method for feature selection for counteracting data skewness on locality sensitive hashing (LSH)-based search may include: (1) ingesting, by an ingestion computer program and from a plurality of data sources, data; (2) extracting, by the ingestion computer program, a plurality of features from the ingested data; (3) transforming, by the ingestion computer program, each of the plurality of features into a feature vector; (4) selecting, by the ingestion computer program, a subset of the plurality of features; and (5) for each selected feature vector: computing, by the ingestion computer program, a random hash function for the selected feature; and inserting, by the ingestion computer program, an output of the random hash function into a hash table with the selected feature.