Network Data Clustering via Hub Node Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods are inadequate for processing large-scale network data, leading to excessive processing time as the number of nodes increases, especially when using community detection methods on systems like Hadoop MapReduce, which become impractically slow for datasets with 0.2 million users or more.
Innovation Solution
A computer processing method and system that filters nodes with a degree higher than a predefined threshold, stores their neighborhood relationships, clusters the filtered data to obtain primary groups, and merges filtered nodes into these groups to reduce processing time and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional community detection methods are used on large-scale network data, then comprehensive analysis of all nodes is achieved, but processing time increases exponentially
Solution Approach 1:
The patent segments the network data processing by dividing nodes into two categories: high-degree nodes (hubs) and ordinary nodes. This segmentation allows the system to apply different processing strategies to different node types, thereby reducing overall processing time while maintaining comprehensive analysis coverage.
Solution Approach 2:
The patent extracts high-degree nodes from the overall network data and processes them separately using a specialized hub node processing module. This extraction allows the system to handle the most computationally intensive part of the data separately, preventing exponential processing time increases while maintaining comprehensive analysis.
2Manufacturing precision
If all nodes are processed in detail, then accurate grouping results are obtained, but processing efficiency decreases
Solution Approach 1:
The patent applies local quality by using different processing approaches for different node types: hub nodes are processed with one level of detail while ordinary nodes are processed with another level. This allows the system to maintain high grouping accuracy for critical hub nodes while processing ordinary nodes more efficiently, thereby improving overall processing efficiency without sacrificing grouping accuracy.
Solution Approach 2:
The patent applies partial action by processing hub nodes in detail while using simplified processing for ordinary nodes. This selective approach ensures that the most important nodes (hubs) receive comprehensive analysis for accurate grouping, while ordinary nodes are processed efficiently, thereby maintaining overall grouping accuracy while improving processing efficiency.
3Reliability
If conventional clustering algorithms are applied to entire network data, then complete community structure is discovered, but computational complexity becomes unmanageable
Solution Approach 1:
The patent applies preliminary action by pre-processing the network data to identify and separate hub nodes before the main clustering process. This preliminary classification allows the subsequent clustering algorithms to operate on a simplified data structure, reducing computational complexity while ensuring that hub nodes are still incorporated into the final community structure for complete detection.
Solution Approach 2:
The patent introduces an intermediary processing layer that handles hub nodes separately before integrating them into the main clustering process. This intermediary module acts as a bridge between the simplified ordinary node processing and the complete community structure detection, reducing computational complexity while maintaining detection completeness.
Data Source
AI summary
A computer processing method includes receiving network data; filtering a node with a degree higher than a predefined threshold in the network data; storing the filtered node and its neighborhood relationship; clustering the filtered network data to obtain primary group(s); and obtaining a final group based on the filtered node and its neighborhood relationship and the primary group(s). The computer processing method and a corresponding system can be applicable to the processing for network data of large scale, and reduce greatly the processing time for clustering the network data of large scale, and the invention can be implemented well in parallel.