Degree-Based Graph Reconstruction for Parallel Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph partitioning technologies in parallel graph processing systems struggle to balance computation load and reduce communication costs effectively, leading to increased execution times of graph processing algorithms.
Innovation Solution
A graph refactorization method that fuses low-degree vertices into super-vertices and splits high-degree vertices into sub-vertices, maintaining the original topological structure while balancing load across machines, thereby reducing excessive communication and shortening execution time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If edge-cut graph partitioning is used to equally divide vertices to all machines, then communication costs are reduced, but load allocation becomes unbalanced and execution time increases
Solution Approach 1:
The patent segments vertices into different types (high-degree, low-degree, medium-degree) and applies different partitioning strategies to each segment. High-degree vertices are kept intact on single machines, low-degree vertices are distributed evenly, and medium-degree vertices are handled with special care. This segmentation resolves the contradiction by allowing communication-efficient handling of low-degree vertices while maintaining load balance through appropriate distribution of high-degree vertices.
Solution Approach 2:
The patent applies different partitioning qualities to different parts of the graph. For low-degree vertices, it uses communication-efficient edge-cut partitioning. For high-degree vertices, it uses load-balancing-oriented vertex-cut partitioning. This local differentiation allows each region of the graph to be partitioned according to its specific characteristics, resolving the universal trade-off between communication cost and load balance.
2Productivity
If vertex-cut graph partitioning is used to evenly allocate edges to all machines, then load allocation is balanced, but low-degree vertices are forcibly split and copied to multiple machines increasing communication costs
Solution Approach 1:
The patent segments vertices by degree and applies vertex-cut partitioning only to high-degree vertices where it provides benefit, while using edge-cut partitioning for low-degree vertices where it would cause unnecessary splitting. This selective application resolves the contradiction by avoiding the harmful effects of vertex-cut on low-degree vertices while retaining its load-balancing benefits for high-degree vertices.
Solution Approach 2:
Instead of applying a single partitioning approach universally, the patent inverts the conventional wisdom by applying edge-cut (typically associated with communication efficiency) to low-degree vertices and vertex-cut (typically associated with load balancing) to high-degree vertices. This inversion resolves the contradiction by matching the partitioning strategy to the vertex characteristics rather than applying a one-size-fits-all approach.
3Productivity
If hybrid-cut is used with different partitioning functions for different vertices, then load is balanced and communication costs are reduced, but coordination costs increase and each machine needs to read more graph information
Solution Approach 1:
The patent segments vertices into three categories and applies different partitioning strategies to each segment, achieving the benefits of hybrid-cut while simplifying the implementation. By clearly defining segments based on degree thresholds and applying appropriate partitioning methods to each, it reduces coordination complexity compared to more sophisticated hybrid approaches while maintaining the dual benefits of load balance and communication efficiency.
4Loss of energy
If graph partitioning is used to reduce communication costs, then coordination between machines is improved, but execution time of graph processing algorithm increases
Solution Approach 1:
The patent performs preliminary classification of vertices by degree before partitioning, and uses this classification information during the partitioning process to make informed decisions about vertex placement. This preliminary action allows the system to optimize both communication costs and execution time simultaneously by pre-identifying which vertices should be kept intact and which can be distributed, avoiding the trade-off between the two objectives.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A graph refactorization method and a graph refactorization apparatus are provided, and relate to the field of communication technologies, to implement parallel graph processing with load balancing, low communication costs, and short execution time. The method includes: A first machine selects a first vertex from a local vertex set of a subgraph (S801). If a degree of the first vertex is less than or equal to a first preset threshold, the first machine fuses the first vertex and an adjacent vertex within a preset quantity of hops of the first vertex in the local vertex set into a super-vertex (S802). If the degree of the first vertex is greater than or equal to a second preset threshold, the first machine copies the first vertex to obtain N sub-vertices, and divides edges of the first vertex in the subgraph of the first machine into N parts to respectively connect to the N sub-vertices (S803).