Dataset Clustering via Segmented Medoid Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for partitioning datasets around medoids scale poorly with increasing dataset size and complexity, requiring significant memory and computations, and are not efficient in handling larger datasets or discrete data types.
Innovation Solution
The proposed methods involve selecting subsets of data to update clusters, using data-to-cluster and data-to-dataset matrices to determine new medoids and costs, allowing for online updates that reduce time and memory usage without sacrificing accuracy, and can handle both discrete and continuous data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing medoid-based partitioning methods are used, then clustering accuracy and robustness to outliers are improved, but computational time and memory requirements increase significantly with dataset size
Solution Approach 1:
The patent divides the dataset into multiple subsets or blocks, processing each subset separately to compute medoids. This segmentation reduces the computational burden on any single processing step while maintaining overall clustering accuracy through iterative refinement across subsets.
Solution Approach 2:
The patent computes medoids using only a subset of data points rather than the entire dataset in each iteration. By selecting representative subsets strategically, the method achieves acceptable clustering accuracy with significantly reduced computational time compared to using all data points.
2Reliability
If existing medoid-based partitioning methods are used, then clustering accuracy and robustness to outliers are improved, but memory requirements increase with the square of the number of data points
Solution Approach 1:
The patent segments the data matrix into smaller blocks that can be stored and processed in memory simultaneously. This block-based approach reduces peak memory requirements while enabling computation on large datasets that would otherwise exceed available memory capacity.
Solution Approach 2:
The patent loads and processes only subsets of the data in memory at any given time rather than loading the entire dataset. This partial action approach reduces memory requirements from O(n²) to O(k·n) where k is the number of blocks, making large-scale clustering feasible.
3Measurement precision
If medoids are computed using all data points, then clustering accuracy is maintained, but the number of calculations increases with the square of the dataset size
Solution Approach 1:
The patent divides the computation into segments processing different subsets of data points. By computing medoids iteratively on segmented data and refining results across segments, the method achieves comparable accuracy to full-data computation with linear rather than quadratic scaling.
Solution Approach 2:
The patent uses partial data subsets to compute medoids in each iteration rather than all data points. This partial action reduces computational complexity from O(n²) to O(k·n) per iteration while maintaining clustering accuracy through multiple refinement passes.
Data Source
AI summary
Computer-implemented methods are provided for analyzing datasets. Consistent with disclosed embodiments, a computing system may be configured to select a cluster from clusters partitioning the dataset. The clusters may include a subset of the dataset, and may be associated with a current medoid of the cluster and a current cost of the cluster. The computing system may determine a new cost of the selected cluster and a new medoid of the selected cluster based on a matrix with rows corresponding to data in a subset of the cluster. The columns may correspond to data in the dataset or only to data in the cluster. The computer system may replace the current medoid of the selected cluster with the new medoid of the selected cluster based on the new cost of the selected cluster. The computer system may output the cluster information to determine a structure of the dataset.