Graph Partitioning Edge Allocation for Distributed Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph computing systems face high communication costs due to frequent synchronization requirements between devices, leading to low efficiency, especially with existing graph partitioning methods that result in high replication factors and excessive communication.
Innovation Solution
A graph partitioning method that extracts edges one by one and allocates them based on aggregation degree and load balancing thresholds, caching edges when conditions are not met to reduce synchronization needs and allocate them efficiently to devices with suitable loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If edges are randomly allocated to different devices, then graph partitioning efficiency is improved, but replication factor increases and communication costs increase
Solution Approach 1:
The patent applies local quality by allocating edges based on their aggregation degree to specific devices. Instead of uniform random allocation, edges with higher aggregation degrees (indicating stronger local connectivity) are allocated to devices that can handle them, creating non-uniform but optimized distribution that reduces replication factor and communication overhead while maintaining partitioning efficiency
2Reliability
If frequent status synchronization is performed to maintain computing status consistency, then computing correctness is ensured, but communication costs increase and computing efficiency decreases
Solution Approach 1:
The patent applies preliminary action by pre-calculating and considering the aggregation degree of edges during the allocation phase. By anticipating which edges will require synchronization based on their aggregation characteristics, the system can pre-plan allocation strategies that minimize future synchronization needs, thereby reducing communication overhead while ensuring correctness
Data Source
AI summary
The method of the present disclosure includes: after a graph partitioning apparatus extracts an edge, first determining whether an aggregation degree between a currently extracted edge and an allocated edge in a first device satisfies a preset condition; then, when the preset condition is satisfied, determining whether a quantity of allocated edges stored in the first device is less than a first preset threshold; and allocating the currently extracted edge to the first device when the quantity is less than the first preset threshold. In this way, an aggregation degree between allocated edges in each device is relatively high and each device has relatively balanced load. When an edge changes and an edge associated with the particular edge needs to be synchronized, a relatively small quantity of devices need to perform synchronization and update, so that costs of communication between devices are reduced, and distributed graph computing efficiency is improved.