Graph-Based Clustering With LSH Bucketing for Large Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Density-based clustering methods face limitations in processing large datasets and datasets with high dimensionality, requiring excessive processing resources and time due to distance querying, leading to bottlenecks and inefficiencies.
Innovation Solution
A graph-based clustering technique using neural network models to predict the number of hash functions based on dataset features, distributing data points into buckets for optimal processing load management, and applying locality-sensitive hashing to increase collision probability, thereby reducing computational burden and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If density-based clustering is applied to large datasets, then clustering analysis can be performed, but processing resources and time required increase excessively
Solution Approach 1:
The patent divides the large dataset into multiple smaller partitions or blocks that can be processed independently and in parallel. This segmentation reduces the computational burden on each processing unit while maintaining the overall clustering analysis capability across the entire dataset.
Solution Approach 2:
The patent introduces a new dimension of processing by implementing distributed computing architecture across multiple nodes or processors. This allows the clustering algorithm to operate in parallel across different data partitions, transforming a single-threaded sequential process into a multi-dimensional parallel processing system.
2Measurement precision
If density-based clustering processes large datasets, then comprehensive pattern identification is achieved, but processing time becomes a limitation
Solution Approach 1:
The patent performs preliminary data preprocessing and partitioning before the main clustering operation. This includes data normalization, feature selection, and initial partitioning that prepares the dataset for faster parallel processing, thereby reducing the overall processing time without compromising pattern identification accuracy.
Solution Approach 2:
The patent implements an iterative refinement process where clustering results are continuously improved across multiple passes. Each pass builds upon previous results, allowing the system to converge to accurate patterns more efficiently than traditional single-pass methods, thereby reducing total processing time.
3Reliability
If density-based clustering is used for suspicious behavior detection, then detection capability is provided, but processing resources are excessive
Solution Approach 1:
The patent applies different processing strategies to different regions of the data space based on local characteristics. High-density regions containing potential suspicious behaviors receive more intensive analysis, while low-density regions use simplified processing, thereby maintaining detection capability while reducing overall resource consumption.
Solution Approach 2:
The patent dynamically adjusts clustering parameters such as density thresholds, neighborhood radius, and minimum point requirements based on the local data distribution. This adaptive parameter adjustment allows the system to maintain high detection reliability for suspicious behaviors while optimizing resource usage by applying stricter parameters only where necessary.
Data Source
AI summary
Systems and methods for identifying patterns in data including determining a set of first attributes for a first dataset corresponding to a plurality of vertices, performing a first join operation based on the set of first attributes, classifying the first dataset to a collection of buckets based on the set of first attributes, each respective bucket including one or more vertices of the plurality of vertices, determining, for a respective bucket, a set of second attributes, performing a second join operation on the set of second attributes to generate a second dataset, and determining a cluster based on applying a first threshold to the second dataset, the cluster including one or more respective vertices in the second dataset exceeding the first threshold. The first threshold corresponding to a minimum number of vertices within a distance parameter of a given vertex.