Parallel Data Clustering via Sample Partitioning and Core Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel processing techniques face challenges in efficiently handling large-scale data clustering due to computational overhead, delays in processing, and impracticality in comparing large datasets with smaller ones, making it difficult to leverage multiple processors effectively.
Innovation Solution
A computer system and method that divides samples and attributes into subsets corresponding to multiple processing cores, allowing for parallel comparison and clustering operations, including similarity comparisons and reassignment of samples, to efficiently form and stabilize clusters across multiple cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If parallel processing is used to speed up data clustering, then processing speed is improved, but computational overhead increases due to data copying and result joining
Solution Approach 1:
The patent segments the large dataset into multiple partitions that can be processed in parallel across different processors. Each processor handles a specific partition, avoiding the need to copy entire datasets while maintaining parallel processing benefits. The segmentation is done in a way that minimizes communication overhead between processors.
Solution Approach 2:
The patent introduces a coordinator processor that acts as an intermediary to manage data distribution and result aggregation. Instead of directly copying data between all processors, the coordinator mediates the data flow, reducing the overall computational overhead associated with parallel processing coordination.
2Measurement precision
If all records are compared with each cluster to ensure accuracy, then clustering precision is improved, but processing time increases significantly
Solution Approach 1:
The patent performs preliminary actions by pre-computing similarity metrics and organizing data in a way that facilitates efficient comparison. Distance matrices are pre-calculated and stored, allowing rapid comparison during the clustering process without requiring exhaustive real-time computations.
Solution Approach 2:
The patent applies partial action by using sampling techniques where not all possible comparisons are performed. Instead, representative samples are used to determine cluster assignments, achieving sufficient clustering precision with reduced processing time. The approach balances completeness with efficiency by performing comparisons only where necessary.
3Productivity
If datasets are divided for parallel processing to utilize multiple processors, then productivity is improved, but coordination complexity and communication overhead increase
Solution Approach 1:
The dataset is segmented into independent partitions that can be processed in parallel with minimal coordination. Each partition is assigned to a specific processor, and the segmentation strategy ensures that data dependencies are minimized, reducing the complexity of coordination between processors.
Solution Approach 2:
The patent merges the processing results from multiple partitions in a systematic way. The coordinator processor combines intermediate results efficiently, using aggregation techniques that reduce communication overhead. This merging approach maintains productivity benefits while managing coordination complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A cluster selector may determine a plurality of sample clusters, and may reproduce the plurality of sample clusters at each of a plurality of processing cores. A sample divider may divide a plurality of samples stored in a database with associated attributes into a number of sample subsets corresponding to a number of the plurality of processing cores, and may associate each of the number of sample subsets with a corresponding one of the plurality of processing cores. A joint operator may perform a comparison of each sample of each sample subset at each corresponding core of the plurality of processing cores with respect to each of the plurality of sample clusters reproduced at the corresponding processing core, based on associated attributes thereof.