Subgraph Segmentation for High-Degree Graph Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale graph computing on heterogeneous many-core systems faces challenges due to unbalanced load distribution, scalability issues, and inefficient data access in traditional partitioning methods, particularly in one-dimensional and two-dimensional partitioning, which leads to excessive communication and storage requirements.
Innovation Solution
A 1.5-dimensional graph partitioning method that segments high-degree vertices into multiple subgraphs, using a hybrid dimension division based on vertex degrees, combined with adaptive traversal directions and optimized data structures like compressed sparse row format and remote memory access, to balance load and minimize communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If one-dimensional vertex partitioning method is used, then vertices are evenly divided into different machines, but heavy vertices will deploy delegates at many nodes causing excessive communication overhead
Solution Approach 1:
The patent segments high-degree vertices into multiple subgraphs based on degree thresholds, creating hierarchical partitioning levels. This segmentation prevents heavy vertices from deploying delegates at too many nodes by limiting their scope to specific subgraphs, thereby reducing communication overhead while maintaining partitioning effectiveness.
Solution Approach 2:
The patent applies different partitioning strategies to different vertex degree categories. Low-degree vertices use one-dimensional partitioning while high-degree vertices are handled through subgraph segmentation with degree-aware thresholds. This local quality approach optimizes communication patterns for each vertex type, reducing overall communication overhead.
2Productivity
If two-dimensional edge-based partitioning method is used, then edges are evenly distributed to achieve load balancing, but too many vertices deploy delegates on rows and columns increasing system complexity
Solution Approach 1:
The patent segments the graph into multiple subgraphs based on vertex degree thresholds, creating a hierarchical structure. This segmentation reduces the number of delegates needed by confining high-degree vertex management to specific subgraphs rather than requiring global two-dimensional delegate deployment across all rows and columns.
Solution Approach 2:
The patent introduces a degree-based dimensional threshold as an additional partitioning dimension beyond traditional row-column two-dimensional partitioning. This degree-aware dimensionality allows the system to balance load effectively while reducing delegate complexity by grouping vertices of similar degree characteristics together.
3Productivity
If traditional partitioning methods are used on heterogeneous many-core systems, then computing can be performed, but unbalanced load distribution and inefficient data access reduce computing performance
Solution Approach 1:
The patent applies degree-aware thresholds to create locally optimized subgraphs that match the heterogeneous many-core system's computational capabilities. By categorizing vertices into different degree levels and processing them through appropriate subgraphs, the system achieves balanced load distribution across cores with different architectures, improving overall computing performance.
Solution Approach 2:
The patent changes the partitioning parameter from uniform vertex or edge distribution to degree-based segmentation with configurable thresholds. This parameter change allows the system to adapt to heterogeneous many-core architectures by distributing workloads according to vertex degree characteristics, achieving better load balance and data access efficiency across diverse computing resources.
4Productivity
If high-degree vertices are processed in traditional methods, then computation can proceed, but excessive communication and storage requirements arise
Solution Approach 1:
The patent segments high-degree vertices into dedicated subgraphs with degree-based thresholds, separating them from low-degree vertex processing. This segmentation reduces communication and storage requirements by confining heavy vertex operations to specific subgraphs, preventing the need for global communication and storage of all vertex data across the entire system.
Solution Approach 2:
The patent extracts high-degree vertices from the general vertex set and processes them through separate subgraphs with specialized handling. This extraction removes the burden of excessive communication and storage requirements from the main computation flow, allowing high-degree vertices to be processed with optimized resource allocation independent of the overall graph size.
Data Source
AI summary
Provided are a distributed parallel computing based graph computing method, a distributed parallel computing system and a computer readable medium. A graph computing method comprises: obtaining data of a graph to be computed, the graph comprising a plurality of vertices and edges; storing the graph according to a compressed sparse row format, and storing the graph in a sparse matrix form, wherein all adjacent edges having a same vertex are continuously stored, and an offset array is used to support an index function thereof; and for subgraphs in which the degrees of both source vertices and target vertices are greater than a predetermined threshold, in forward graphs, performing segmentation according to target vertices, i.e., the column of a matrix, and in reverse graphs, performing segmentation according to source vertices, i.e., the row of the matrix, and further dividing the subgraph into a plurality of segmented subgraph (SSG) components, so that a target vertex in each SSG is limited within a preset range. According to the SSG method of the embodiment for a high-degree vertex, the conventional problem that the space-time locality of accessing data of an adjacent vertex is poor due to a large vertex range of a single large graph is avoided or relieved.