Distributed Graph Learning Data Fusion via Mirror Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed graph learning, existing methods face inefficiencies in fusing neighbor information across large-scale graph data, leading to communication and calculation waiting issues that reduce learning efficiency.
Innovation Solution
A data fusion method and apparatus that utilize mirror nodes and independent threads for parallel processing, where mirror nodes on different devices are used to aggregate and update representation vectors of graph nodes, allowing for concurrent processing and reduced waiting times through a multi-threaded approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed graph learning is used to process large-scale graph data, then the system can handle billion-level nodes, but devices need to interact with each other to fuse neighboring node information, causing communication overhead and waiting time
Solution Approach 1:
The patent segments the graph data by distributing nodes across multiple devices, with each device maintaining local graph nodes and their neighboring nodes. This segmentation enables parallel processing of different node subsets while reducing the communication burden on individual devices.
Solution Approach 2:
The patent performs preliminary action by pre-fetching and caching neighboring node information locally on each device before the actual fusion operation. Devices prepare and store required data in advance, reducing the need for real-time communication during the fusion process and minimizing waiting time.
2Measurement precision
If neighboring node information is fused into each node's information to consider mutual impact, then graph learning accuracy improves, but devices need to interact with each other, reducing processing efficiency
Solution Approach 1:
The patent introduces an intermediary mechanism where a central coordinator or message queue system manages the exchange of neighboring node information between devices. This intermediary orchestrates the fusion process, allowing devices to obtain required information without direct peer-to-peer interactions, thus maintaining accuracy while improving efficiency.
Solution Approach 2:
The patent implements continuous useful action by overlapping communication and computation phases. While devices are computing fusion results for current nodes, they simultaneously receive and prepare neighboring node information for subsequent nodes, ensuring that processing continues without idle waiting periods.
3Quantity of substance
If graph data is segmented and stored on multiple devices for distributed learning, then large-scale graph processing becomes feasible, but coordination between devices increases complexity
Solution Approach 1:
The patent applies universality by designing a standardized interface and communication protocol that all devices in the distributed system must follow. This universal framework enables any device to participate in the distributed learning process with the same rules for data exchange, fusion operations, and coordination, simplifying the overall system complexity despite the multi-device architecture.
Data Source
AI summary
Embodiments of this specification provide a data fusion method for distributed graph learning, applied to a distributed graph learning process for graph data in a distributed system. A single device in the distributed system is pre-allocated with a plurality of graph nodes of the graph data and a corresponding node connection relationship, a first device includes N graph nodes and M mirror nodes, a single mirror node and a single graph node in the N graph nodes are neighboring nodes of each other, and in a data fusion process for distributed graph learning, the first device performs a fusion operation on each of the M mirror nodes through a plurality of mutually independent mirror fusion threads, and separately adds a mirror fusion vector of a mirror node to a local aggregation data sequence. In addition, the first device sequentially sends, through a sending thread, mirror fusion vectors.