Hash Partitioning for Sparse Data Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data-clustering systems are inefficient in processing large and sparse datasets, requiring excessive computing resources and time, and lack flexibility to adjust clusters in real-time or near-real time, making them impractical for dynamic data environments.
Innovation Solution
The dynamic clustering system uses hash partitioning to determine local densities and distances among data points, allowing for real-time clustering by generating hash signatures, mapping data points to hash partitions, and applying a density-peaks-clustering algorithm based on these partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional data-clustering systems use point-by-point comparison to cluster datasets, then clustering accuracy is improved, but computing resources and processing time increase significantly
Solution Approach 1:
The patent divides the dataset into multiple hash partitions based on hash signatures of data points. This segmentation allows the system to process only data points within the same partition rather than comparing every point to every other point, significantly reducing computing resources while maintaining clustering accuracy through the density-peaks algorithm applied within each partition.
Solution Approach 2:
The patent introduces hash partitions as an intermediary structure between the raw data and the clustering process. Hash signatures serve as intermediaries to group similar data points together, enabling the density-peaks algorithm to operate more efficiently by working with pre-organized data rather than raw unsorted data, thus reducing computational complexity.
2Measurement precision
If conventional data-clustering systems use point-by-point comparison to cluster datasets, then clustering accuracy is improved, but processing time increases significantly
Solution Approach 1:
The patent segments the dataset into hash partitions, allowing the density-peaks algorithm to process data in smaller, manageable groups rather than the entire dataset at once. This segmentation dramatically reduces processing time while maintaining accuracy by ensuring that clustering decisions are made based on local density characteristics within each partition.
Solution Approach 2:
The patent performs preliminary action by computing hash signatures and creating hash partitions before applying the density-peaks clustering algorithm. This pre-processing step organizes the data in advance, eliminating the need for time-consuming point-by-point comparisons during the actual clustering process and enabling faster real-time or near-real-time clustering.
3Productivity
If conventional data-clustering systems process datasets, then clustering can be performed, but the system lacks flexibility to adjust clusters in real-time or respond to data changes
Solution Approach 1:
The patent implements a dynamic clustering system where hash partitions can be updated in real-time as new data points are added or existing data changes. The density-peaks algorithm can re-execute on modified partitions without requiring complete reprocessing of the entire dataset, enabling the system to adapt to changing data characteristics and provide real-time or near-real-time clustering adjustments.
4Quantity of substance
If conventional data-clustering systems use complex processing to handle sparse datasets, then data completeness is improved, but computing resources and processing time increase
Solution Approach 1:
The patent segments sparse datasets into hash partitions, which allows the system to handle sparsity more effectively by processing only relevant data points within each partition. This segmentation reduces the computational burden of handling sparse data compared to processing the entire sparse matrix, while still maintaining data completeness through systematic coverage of all data points.
Solution Approach 2:
The patent performs preliminary action by computing hash signatures for all data points before applying the density-peaks algorithm. This pre-processing step creates an organized structure that simplifies subsequent processing of sparse data, reducing the computing resources needed to handle sparsity while maintaining complete data representation through the hash partition structure.
Data Source
AI summary
The present disclosure describes systems, non-transitory computer-readable media, and methods for utilizing hash partitions to determine local densities and distances among users (or among other represented data points) for clustering sparse data into segments. For instance, the disclosed systems can generate hash signatures for users in a sparse dataset and can map users to hash partitions based on the hash signatures. The disclosed systems can further determine local densities and separation distances for particular users (or other represented data points) within the hash partitions. Upon determining local densities and separation distances for datapoints from the dataset, the disclosed systems can select a segment (or cluster of data points) grouped according to a hierarchy of a clustering algorithm, such as a density-peaks-clustering algorithm.