Graph Data Processing for K-Core Mining in Large Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current K-core mining algorithms for large-scale networks are inefficient due to recursive pruning methods that consume excessive iteration time and computing resources on non-critical nodes, leading to poor performance in ultra-large-scale networks.
Innovation Solution
A graph data processing method that splits the network graph into dense and sparse subgraphs based on a preset threshold, determining stable nodes and coreness in the dense subgraph, and then processing the sparse subgraph to reduce unnecessary iterations and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If recursive pruning method is used to mine K-core from network graph, then all nodes can be processed systematically, but computation time and memory consumption increase excessively on ultra-large-scale networks
Solution Approach 1:
The network graph is segmented into dense subgraphs and sparse subgraphs based on node degree thresholds. Dense subgraphs contain nodes with high degrees that are more likely to be stable nodes, while sparse subgraphs contain nodes with low degrees. This segmentation allows the algorithm to focus computational resources on the dense subgraphs where stable nodes are more likely to exist, reducing overall computation time while maintaining accuracy.
Solution Approach 2:
Different processing strategies are applied to different parts of the network graph. The dense subgraph processing uses iterative pruning to identify stable nodes with coreness greater than the threshold, while the sparse subgraph processing uses a simplified approach since nodes there are less likely to be stable. This local quality differentiation optimizes the balance between accuracy and computation time.
2Measurement precision
If recursive pruning method processes all nodes iteratively, then comprehensive coreness calculation is achieved, but computing resources are wasted on non-critical nodes
Solution Approach 1:
The algorithm extracts and processes the dense subgraph separately from the sparse subgraph. By taking out the dense subgraph containing potential stable nodes and processing it with iterative pruning, the algorithm avoids wasting computing resources on the sparse subgraph where stable nodes are unlikely to exist, while still ensuring comprehensive coreness calculation for all nodes.
Solution Approach 2:
The algorithm applies iterative pruning excessively to the dense subgraph to ensure all stable nodes are identified with coreness greater than the threshold, while applying a simplified approach to the sparse subgraph. This partial application of the full pruning process optimizes resource usage while maintaining calculation completeness.
3Reliability
If network graph is processed as a whole, then all node relationships are considered, but memory consumption becomes prohibitive for ultra-large-scale networks
Solution Approach 1:
The network graph is segmented into dense and sparse subgraphs based on node degree distributions. This segmentation divides the large graph into smaller, more manageable components that can be processed independently with reduced memory requirements, while still considering all node relationships through the systematic processing of both subgraphs.
Solution Approach 2:
Different memory management strategies are applied to different subgraphs. The dense subgraph is processed with full iterative pruning that considers all local relationships, while the sparse subgraph is processed with a simplified approach. This local quality differentiation maintains relationship analysis accuracy for critical nodes while reducing overall memory consumption.
Data Source
AI summary
Provided is a graph data processing method, including: acquiring a degree of association of each node in a network graph; splitting the network graph to obtain a dense subgraph according to the degree of association of each node; determining stable nodes in the network graph and coreness of the stable nodes based on the dense subgraph, the coreness of the stable nodes being greater than a preset threshold; obtaining a sparse subgraph in the network graph according to the preset threshold, and remaining nodes in the network graph other than the stable nodes and connecting edges between the remaining nodes; and determining the coreness of each node in the sparse subgraph based on the sparse subgraph and the stable nodes.