Incremental Data Clustering Across Batches With Predefined Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Clustering large datasets is challenging due to high memory and time complexity, and existing algorithms lack flexibility to continue clustering with pre-existing clusters, often requiring reformation from scratch or predicting cluster numbers.
Innovation Solution
A distance-based clustering method that allows clustering new data points within pre-defined clusters, using techniques like Euclidean, Manhattan, and Jaccard distances, and supports flexible addition or merging of samples into existing clusters, reducing memory and time complexity while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If clustering algorithms are used to handle large datasets, then meaningful groups of entities can be found, but memory and time complexities increase significantly
Solution Approach 1:
The patent divides the large dataset into multiple smaller batches and processes them separately. Each batch is clustered independently or with reference to previously processed batches, avoiding the need to load and process the entire large dataset at once. This segmentation reduces memory requirements and allows incremental processing, thereby reducing execution time while maintaining clustering quality.
2Measurement precision
If clustering algorithms are used to handle large datasets, then meaningful groups of entities can be found, but memory requirements increase significantly
Solution Approach 1:
The dataset is segmented into manageable batches that can be processed in memory without requiring the entire dataset to be loaded simultaneously. This approach reduces peak memory usage while still allowing comprehensive clustering analysis across all data points through sequential or iterative processing.
Solution Approach 2:
The patent performs preliminary clustering on the first batch to establish initial clusters before processing subsequent batches. This preliminary action creates a foundation that guides the clustering of remaining data, reducing the computational burden and memory requirements for processing the entire dataset.
3Adaptability or versatility
If existing clustering algorithms are used, then clusters can be formed, but flexibility to continue clustering with pre-existing clusters is lacking
Solution Approach 1:
The algorithm performs preliminary clustering on the first batch to establish initial clusters. These pre-existing clusters serve as a foundation for subsequent batches, allowing the system to continue clustering incrementally without reformatting from scratch. This approach provides flexibility to process data in stages while maintaining cluster consistency.
Solution Approach 2:
The patent enables continuous clustering by allowing subsequent batches to be clustered with reference to previously formed clusters. This continuity allows the clustering process to be extended incrementally as new data becomes available, rather than requiring complete re-clustering, thereby improving adaptability while managing complexity through iterative refinement.
Data Source
AI summary
Method (400) and system for clustering data samples is disclosed. The method (400) may include receiving (402) a plurality of batches, each of the plurality of batches including a plurality of samples, and creating (404) a set of clusters from a first batch of the plurality of batches, using a clustering technique. Each cluster of the set of clusters may include one or more samples. The one or more samples are determined based on variability of the one or more samples and cluster attributes associated with each cluster. The method may further include reconfiguring (406) the set of clusters that includes at least one of: repopulating (400A) the existing set of clusters with samples from remaining batches of the plurality of batches; and adding (400B) a new set of clusters to the existing set of clusters and populating the new set of clusters with samples from the remaining batches.