Parallel Data Clustering via Segmentation and Co-clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data clustering methods face a tradeoff between effectiveness and scalability, struggling to handle large and highly dimensional datasets efficiently, leading to resource-intensive processes or low-quality results.
Innovation Solution
A parallelized data clustering method that incorporates information-theoretic co-clustering, breaking down global optimization into local optimizations executable in parallel, utilizing a master processor and slave processors to optimize objective functions across cluster pairs, allowing for scalable and powerful clustering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If powerful clustering algorithms are used, then clustering effectiveness is improved, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent divides the dataset into multiple partitions and assigns them to different processing units. Each partition is clustered independently using powerful algorithms, then results are merged. This segmentation allows effective clustering on manageable subsets without overwhelming system resources, resolving the contradiction between algorithm power and processing efficiency.
Solution Approach 2:
The patent introduces parallel processing as an additional dimension, transforming sequential execution of powerful algorithms into concurrent execution across multiple processing units. This dimensional shift enables the system to maintain high clustering effectiveness while achieving linear scalability in processing capacity.
2Productivity
If simple clustering algorithms are used, then processing speed and scalability are improved, but clustering quality deteriorates
Solution Approach 1:
The patent merges multiple simple parallel clustering operations into a unified result by combining cluster assignments from different partitions. This merging strategy preserves the scalability and speed advantages of simple algorithms while achieving quality comparable to powerful algorithms through the aggregation of multiple independent clustering results.
3Quantity of substance
If clustering is performed on large datasets with many dimensions, then comprehensive data coverage is improved, but resource requirements and processing time explode
Solution Approach 1:
The patent segments both the data and the processing workload, dividing large datasets into smaller partitions that can be handled with moderate resources. Each partition is processed independently, allowing comprehensive coverage of the entire dataset while keeping per-unit resource consumption manageable. The segmented approach prevents resource explosion while maintaining complete data coverage.
Data Source
AI summary
A method for data clustering may comprise entering data into a computer network comprising a master processor, an array of slave processors, and two cluster seats associated with each slave processor; executing a master process comprising dividing the data into clusters, sending the clusters to the cluster seats, initializing an optimization cycle, and computing an objective function. The optimization cycle includes the parallel execution by the slave processors of a slave process, which includes exchanging data between paired clusters so as to increase the objective function based on two modalities, and then resorting the cluster pairs for a subsequent iteration of the process.