Density-Based Clustering Radius Search for Large-Scale Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Density-based clustering algorithms face challenges in parameter tuning, particularly in high-dimensional and large-scale datasets, leading to over-segmentation or under-segmentation, which affects the quality and interpretability of clustering results.
Innovation Solution
A modified ternary search algorithm is employed to efficiently tune density-based clustering algorithms by leveraging the near-unimodality of the neighborhood radius parameter, using initial bounds that are closer to the optimal value, reducing computational time and resources while achieving the desired number of clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional parameter tuning methods are used for density-based clustering, then the algorithm can handle arbitrary shapes and sizes, but the computational time and resources increase significantly
Solution Approach 1:
The patent applies preliminary action by performing ternary search to find optimal parameter bounds before executing the main density-based clustering algorithm. The system pre-calculates upper and lower bounds for the neighborhood radius parameter using sampled data, which narrows the search space and accelerates the subsequent clustering process while maintaining high clustering quality
Solution Approach 2:
The patent utilizes parameter changes by dynamically adjusting the neighborhood radius parameter within calculated bounds during the ternary search process. The system modifies parameters such as sampling rates and parameter bounds based on data characteristics, enabling efficient adaptation to different datasets while reducing computational overhead
2Measurement precision
If exhaustive search is used to find optimal parameter values, then clustering accuracy improves, but computational resources are excessively consumed
Solution Approach 1:
The patent applies segmentation by dividing the parameter search space into three equal segments using ternary search. Instead of exhaustively searching the entire parameter space, the algorithm segments it and systematically eliminates two-thirds of the search space in each iteration, achieving high parameter tuning accuracy with significantly reduced computational resources
Solution Approach 2:
The patent uses copying by working with sampled copies of the dataset rather than the complete dataset during parameter tuning. The system creates sampled representations of the data to estimate optimal parameter bounds, which then guide the clustering process on the full dataset, reducing computational resource requirements while maintaining accuracy
3Productivity
If the neighborhood radius parameter is not properly tuned, then the algorithm runs faster, but over-segmentation or under-segmentation occurs
Solution Approach 1:
The patent implements feedback by using the results of sampled clustering runs to inform and adjust the parameter search bounds. The system evaluates clustering outcomes at different parameter values and uses this feedback to refine the upper and lower bounds, ensuring optimal parameter selection that balances clustering speed and accuracy
Solution Approach 2:
The patent applies dynamics by making the parameter bounds adaptive rather than static. The upper and lower bounds for the neighborhood radius parameter are dynamically adjusted based on data characteristics and sampling results, allowing the algorithm to optimize both speed and accuracy for different datasets and clustering scenarios
Data Source
Figure 1
Figure 2A
Figure 2B~2C
AI summary
Systems and methods are provided for implementing large-scale density-based clustering functionalities. In examples, a system selects, for a dataset (which may be sampled at 100 % or less), an upper bound value and a lower bound value of a neighborhood radius parameter of a density-based clustering algorithm. The system identifies, using a modified ternary search algorithm, an optimal neighborhood radius parameter value, based on the upper and lower bound values, outputs the optimal neighborhood radius parameter value and/or a corresponding optimal number of clusters within the dataset. The modified ternary search algorithm leverages the near-unimodality of the neighborhood radius parameter, while selection of the upper bound value leverages a characteristic in which the neighborhood radius parameter value increases as the sampling rate decreases, and selection of the lower bound value uses ternary search that takes the number of clusters as a parameter instead of the neighborhood radius parameter.