Locality Sensitive Hashing for Large Scale Graph Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The efficient construction of informative similarity graphs for extremely large datasets is challenging due to computational burdens, and traditional sampling techniques may miss important similarities between data points.
Innovation Solution
The method leverages locality sensitive hashing (LSH) as an adaptive partition of the input dataset to sample both diverse and highly similar data points, reducing computational requirements while producing accurate graph representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional sampling techniques are used to reduce computational burden, then processing time and memory usage decrease, but important similarities between data points are missed
Solution Approach 1:
The patent segments the large dataset into multiple partitions and applies locality-sensitive hashing to create hash buckets within each partition. This segmentation allows the system to process data in manageable chunks while preserving similarity relationships through the hashing mechanism, thus reducing processing time without losing important similarity information.
Solution Approach 2:
The patent introduces locality-sensitive hashing as an intermediary mechanism between the raw data and the sampling process. The hash functions act as mediators that map data points to hash buckets in a way that preserves similarity relationships, enabling efficient sampling while maintaining the integrity of similarity information.
2Power
If LSH is used to partition and sample data, then computational requirements are reduced, but the complexity of the hashing and sampling process increases
Solution Approach 1:
The patent divides the complex LSH process into manageable segments: data partitioning, hash function application, bucket formation, and sampling. Each segment handles a specific aspect of the process, making the overall complex task more tractable and easier to implement while still achieving the goal of reducing computational power requirements.
3Measurement precision
If more data points are sampled to maintain accuracy, then graph representation accuracy improves, but memory requirements increase
Solution Approach 1:
The patent employs self-service sampling where the locality-sensitive hashing mechanism automatically identifies and selects representative data points based on their similarity relationships. The hash buckets naturally concentrate similar points together, allowing the sampling process to automatically select an optimal subset that maintains accuracy without requiring manual intervention or excessive memory usage.
Data Source
AI summary
A method and systems are disclosed for data sampling using locality sensitive hashing. Training data set comprising a plurality of data points is received. Each data point of the plurality of data points is assigned to a hash bucket of a set of hash buckets associated with a set of hash functions. A sample set of data points is generated by sampling data points from each bucket of the set of hash buckets. Each sample data point pair comprises a pair of data points from the sample set of data points. An artificial intelligence (AI) model to output a numerical value that produces a degree of similarity between an input pair of data points is trained using the plurality of sample data point pairs. A data structure representing relationships between data points of the plurality of data points is generated using the trained AI model and the training data set.