Neural Network Graph Partitioning for Low-Overhead Parallel Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks processing can be slow or infeasible in all devices due to inefficient parallel computation schemes.
Innovation Solution
A device and method for determining a parallel computation scheme by transforming a computation graph into a dataflow graph with recursive subgraphs, selecting partitioning axes based on communication costs, and ensuring compliance with memory requirements, enabling efficient and deterministic parallel computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If neural network computation is performed on all devices, then processing capability is improved, but processing speed deteriorates due to inefficiency
Solution Approach 1:
The computation graph is transformed into a dataflow graph and divided into multiple recursive subgraphs through partitioning recursions. Each subgraph can be processed independently on different computing devices, enabling parallel computation while maintaining the overall processing capability of the neural network.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by transforming the sequential computation graph into a multi-dimensional dataflow graph structure. This allows computations to be executed across multiple dimensions (devices, subgraphs, operations) simultaneously, dramatically improving processing speed without sacrificing capability.
2Productivity
If parallel computation is implemented, then processing speed is improved, but communication overhead increases
Solution Approach 1:
The patent performs preliminary actions by determining the processing order of recursive subgraphs before actual computation begins. The partitioning scheme and dataflow graph are pre-established, which optimizes communication patterns in advance and reduces the communication overhead during actual parallel execution.
Solution Approach 2:
The dataflow graph serves as an intermediary structure between the computation graph and the parallel execution on multiple devices. It mediates the data flow and communication requirements, allowing efficient coordination between computing devices while minimizing communication overhead.
3Productivity
If computation graph is transformed into dataflow graph with recursive subgraphs, then parallel computation efficiency is improved, but system complexity increases
Solution Approach 1:
The computation graph is transformed into a nested structure of recursive subgraphs, where each subgraph can contain further nested subgraphs. This nested organization allows the complex system to be broken down into manageable hierarchical levels, improving parallel computation efficiency while making the complexity structured and controllable.
Data Source
AI summary
A device receives a computation graph and transforms the computation graph into a dataflow graph comprising recursive subgraphs. Each recursive subgraph comprises a tuple of another recursive subgraph and an operator node, or an empty graph. The device determines a number of partitioning recursions based on a number of parallel computing devices. For each partitioning recursion, the device determines costs corresponding to operator nodes, determines a processing order of the recursive subgraphs, and processes the recursive subgraphs. To process a recursive subgraph, the device selects a partitioning axis for tensors associated with an operator node of the recursive subgraph. The device outputs a partitioning scheme comprising partitioning axes for each tensor associated with the operator nodes.