Computational Graph Partitioning With Send-Receive Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require significant processing time due to the large number and size of operations, which can be exacerbated when distributed across multiple devices with varying hardware profiles, leading to inefficient communication and coordination burdens.
Innovation Solution
Modifying computational graphs by inserting send and receive nodes to compartmentalize operations across devices, allowing devices to communicate independently and reducing the need for centralized coordination, thus optimizing distributed processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network operations are distributed across multiple devices, then processing throughput is improved, but communication coordination overhead increases
Solution Approach 1:
The patent introduces send and receive nodes as intermediary elements that mediate communication between devices. These nodes act as standardized interfaces that simplify the coordination process by providing a uniform mechanism for data transfer, reducing the complexity of direct peer-to-peer communication coordination between multiple devices.
Solution Approach 2:
The computational graph is segmented into subgraphs that can be independently allocated to different devices. Each subgraph is self-contained with explicit send and receive nodes that define communication boundaries, allowing devices to process segments independently while minimizing coordination overhead through clear interface definitions.
2Loss of time
If computational graph is partitioned across devices, then processing time is reduced, but network communication requirements increase
Solution Approach 1:
The send and receive nodes are inserted during the graph partitioning phase, before execution. This preliminary structuring of communication interfaces allows the system to optimize data transfer paths and anticipate communication needs, reducing actual runtime network communication costs while maintaining parallel processing benefits.
3Adaptability or versatility
If send and receive nodes are inserted for each cross-device edge, then device communication independence is improved, but computational graph complexity increases
Solution Approach 1:
The patent applies homogeneity by using uniform send and receive node structures for all cross-device communication edges. This standardized approach allows devices to communicate independently using the same interface pattern, improving adaptability while managing graph complexity through repetition of proven, simple communication primitives rather than custom communication logic for each device pair.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for modifying a computational graph to include send and receive nodes. Communication between unique devices performing operations of different subgraphs of the computational graph can be handled efficiently by inserting send and receive nodes into each subgraph. When executed, the operations that these send and receive nodes represent may enable pairs of unique devices to conduct communication with each other in a self-sufficient manner. This shifts the burden of coordinating communication away from the backend, which affords the system that processes this computational graph representation the opportunity to perform one or more other processes while devices are executing subgraphs.