Graph Data Partitioning with Mirror Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently partitioning large-scale graph data across multiple computing nodes for distributed graph computing, as conventional single-machine schemes are inadequate, is addressed by combining node partitioning with edge data allocation and maintaining mirror graph nodes to reduce communication costs.
Innovation Solution
Graph data partitioning is performed by partitioning graph nodes based on degrees using a computing load balancing algorithm, allocating edge data to corresponding partitions, and constructing mirror graph nodes for efficient distribution across computing nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If graph data is partitioned across multiple computing nodes for distributed graph computing, then computing efficiency is improved, but communication costs across partitions increase
Solution Approach 1:
The patent segments graph data into multiple partitions distributed across different computing nodes. Each partition contains a subset of graph nodes and their associated edge data, enabling parallel processing and improving computing efficiency for large-scale graph data.
Solution Approach 2:
The patent creates mirror graph nodes as copies of original graph nodes and places them in the same partition as their associated edge data. This copying approach eliminates the need for cross-partition communication during graph computations, thereby reducing communication costs while maintaining distributed processing benefits.
2Device complexity
If graph nodes are partitioned without considering edge data location, then node distribution is simplified, but communication costs increase due to cross-partition edge access
Solution Approach 1:
The patent merges graph nodes with their associated edge data into the same partition by creating mirror graph nodes. This combination ensures that both nodes and edges are co-located within the same partition, eliminating cross-partition edge access and reducing communication costs.
Solution Approach 2:
The mirror graph nodes act as intermediaries that enable local access to edge data within the same partition. Instead of directly accessing edges in remote partitions, computing nodes use mirror graph nodes as local proxies, thereby avoiding cross-partition communication.
3Loss of energy
If mirror graph nodes are added to partitions, then communication costs are reduced, but data storage requirements increase
Solution Approach 1:
The patent applies local quality by creating mirror graph nodes only where needed - specifically in partitions that contain incoming edges from external partitions. Not all partitions require mirror nodes, and the mirroring is selectively applied based on the partition's edge data characteristics, optimizing storage efficiency.
Solution Approach 2:
The patent changes the storage parameter by storing only the necessary graph node data in mirror form rather than duplicating entire graph partitions. The mirror graph nodes contain only the essential node attributes needed for local computation, reducing the storage overhead compared to full partition duplication.
Data Source
AI summary
Method, apparatus and computer-readable media are provided. During graph data partitioning, graph nodes in graph data are partitioned based on degrees of the graph nodes according to a computing load balancing allocation algorithm, such that the graph nodes are partitioned as primary graph nodes into graph data partitions. Subsequently, edge data of associated edges of the primary graph nodes are allocated to corresponding graph data partitions, where the associated edges include outgoing edges and/or incoming edges. Additionally, for an associated edge of a primary graph node, a replica of another graph node that corresponds to the primary graph node for the associated edges is constructed to be stored as a mirror graph node in the graph data partition corresponding to the primary graph node.