Trimming Oversized Data Clusters via Breadth-First Search Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for synchronizing large datasets suffer from inefficiencies, particularly in breaking down large clusters due to inaccurate data entry, leading to excessive computational time and memory usage when applying rudimentary business rules or hierarchal clustering algorithms.
Innovation Solution
A technique involving a breadth-first search (BFS) is employed to segment oversized clusters into smaller sub-clusters, improving processing time and reducing memory requirements by generating new clusters when the BFS exceeds a predetermined size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rudimentary business rules or hierarchal clustering algorithms are applied to break down large clusters, then clustering accuracy is improved, but computational time and memory usage increase significantly
Solution Approach 1:
The patent segments the large cluster breaking task into multiple passes, where each pass processes a subset of records. Instead of applying hierarchal clustering to all records at once, the system divides the work into manageable chunks, processing them sequentially to reduce computational complexity from O(n³) to a more scalable approach.
Solution Approach 2:
The patent applies partial action by using a sampling approach where not all record pairs are compared in every pass. Instead of performing complete pairwise comparisons across the entire dataset, the system performs partial comparisons on subsets, achieving reasonable clustering accuracy with significantly reduced computational effort.
2Measurement precision
If hierarchal clustering algorithm is applied to large clusters, then clustering accuracy is improved, but memory requirements increase to O(n²)
Solution Approach 1:
The patent segments the data processing into multiple passes, where each pass works with a subset of records rather than the entire dataset. This segmentation allows the system to maintain only the necessary data structures for the current pass in memory, avoiding the O(n²) memory requirement of traditional hierarchal clustering.
Solution Approach 2:
The system performs partial clustering operations on subsets of records in each pass, rather than attempting to cluster all records simultaneously. This partial action approach allows the algorithm to achieve progressive clustering results while keeping memory usage bounded by the subset size rather than the square of the total dataset size.
3Reliability
If pairwise comparisons are performed across all record pairs in a large cluster, then clustering completeness is improved, but time complexity increases to O(n²)
Solution Approach 1:
The patent segments the pairwise comparison task into multiple passes, where each pass compares records within a subset rather than all records against all other records. This segmentation reduces the time complexity from O(n²) for complete pairwise comparison to a more manageable level by processing subsets sequentially.
Solution Approach 2:
The patent maintains clustering completeness through multiple continuous passes that progressively refine the clustering. Each pass continues the useful action of comparing and clustering records, building upon the results of previous passes to achieve complete and accurate clustering without requiring a single exhaustive O(n²) comparison step.
Data Source
AI summary
Disclosed are techniques for trimming large clusters of related records. In one embodiment, a method is disclosed comprising receiving a set of clusters, each cluster in the clusters including a plurality of records. The method extracts an oversized cluster in the set of clusters and performs a breadth-first search (BFS) on the oversized cluster to generate a list of visited records. The method terminates the BFS upon determining that the size of the list of visited records exceeds a maximum size and generates a new cluster from the list of visited records and adding the new cluster to the set of clusters. By recursively performing BFS traverse over the oversized cluster and extracting smaller new clusters from it, the oversized cluster is eventually partitioned into a set of sub-clusters with the size smaller than the predefined threshold.