Graph Data Partitioning by Vertex Features for Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph data partition methods in distributed clusters result in uneven computational loads and excessive communication overheads due to the lack of consideration for vertex relationships, leading to inefficient graph computation.
Innovation Solution
A method and apparatus for partitioning graph data into datasets based on local features representing vertex proximity, ensuring balanced computational loads and reducing inter-node communications by partitioning vertices and edges accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If graph data is partitioned without considering vertex relationships, then partitioning is simple, but computational loads become uneven and communication overheads increase
Solution Approach 1:
The patent applies preliminary action by pre-calculating vertex features (degree, betweenness centrality, clustering coefficient) and using them to guide the partitioning process. This preliminary analysis of vertex relationships enables the partitioning algorithm to make informed decisions about data distribution, thereby achieving balanced computational loads and reduced communication overhead while maintaining partitioning efficiency
Solution Approach 2:
The patent changes the partitioning parameters by incorporating vertex features (degree, betweenness centrality, clustering coefficient) as additional dimensions for partitioning decisions. Instead of using simple hash-based or random partitioning, the system adjusts partitioning strategies based on these vertex parameters, enabling balanced data distribution that considers the underlying graph structure and relationships
2Speed
If vertices are partitioned without considering local features, then partitioning is faster, but communication overheads between nodes increase
Solution Approach 1:
The patent applies local quality by partitioning vertices based on their local structural features such as degree, betweenness centrality, and clustering coefficient. Vertices with similar local characteristics are grouped together in the same partition, which preserves local connectivity patterns and reduces the need for cross-node communication during graph computations, thereby reducing communication overhead while maintaining partitioning efficiency
3Device complexity
If edges are partitioned without considering target vertices, then partitioning is simpler, but the quantity of communications between nodes increases
Solution Approach 1:
The patent applies merging by combining the partitioning of vertices and edges into a unified strategy where edges are assigned to partitions based on their target vertices. This ensures that both the source and target vertices of an edge are located in the same partition or adjacent partitions, reducing the need for cross-node communication and information exchange during graph computations
Data Source
AI summary
Embodiments of this specification disclose graph data partition computer-implemented methods, non-transitory, computer-readable media, and computer-implemented systems. A computer-implemented method includes partitioning vertices in graph data into a plurality of dataset. Edges in the graph data are partitioned into datasets that include target vertices of the edges, where the datasets are used by nodes in a distributed cluster to perform graph computation, and where computational loads of the plurality of datasets are similar. Implementations of this specification can achieve load balancing between nodes in the distributed cluster and can reduce communication overhead.