Neural Network Partitioning Axes for Deterministic Parallel Computing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks processing can be slow or infeasible in all devices due to inefficient and non-deterministic parallel computation schemes.
Innovation Solution
A method for determining a parallel computation scheme for neural networks involves transforming a computation graph into a recursive dataflow graph, selecting partitioning axes for tensors based on communication costs, and dividing tensors into subtensors for parallel processing by multiple computing devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If neural networks are processed on all devices, then automation capability is improved, but processing speed deteriorates due to inefficient parallel computation schemes
Solution Approach 1:
The computation graph is segmented into operator nodes and tensors that can be independently processed. Each operator node represents a discrete computational unit, and tensors are divided into subtensors along partitioning axes, enabling parallel processing across multiple devices while maintaining automation capability
Solution Approach 2:
The patent introduces a new dimension of parallelism by creating recursive dataflow graphs that operate across multiple computational dimensions. This allows the system to process neural networks in parallel across devices, layers, and operations simultaneously, transforming sequential processing into multi-dimensional parallel execution
2Productivity
If parallel computation is implemented without deterministic schemes, then processing capability is improved, but reliability deteriorates due to non-deterministic behavior
Solution Approach 1:
The system performs preliminary actions by determining partitioning schemes and processing orders before actual computation begins. The recursive dataflow graph structure pre-defines how computations will be distributed and ordered, ensuring deterministic behavior while enabling parallel processing capability
Solution Approach 2:
The patent incorporates feedback mechanisms through cost determination and processing order optimization. The system evaluates computational costs and communication requirements, then adjusts partitioning strategies to achieve both high productivity and deterministic outcomes through iterative optimization
3Productivity
If tensors are divided into subtensors for parallel processing, then productivity is improved, but communication overhead increases
Solution Approach 1:
The patent applies local quality by determining communication costs specific to each operator node and tensor partitioning decision. Instead of uniform partitioning, the system optimizes subtensor distribution based on local communication requirements, placing computationally intensive operations with high communication needs strategically to minimize overall overhead
Solution Approach 2:
The system dynamically changes partitioning parameters such as partitioning axes and subtensor dimensions based on communication cost analysis. By adjusting these parameters according to operator-specific requirements, the system achieves high parallel productivity while minimizing communication overhead through optimized data distribution
Data Source
Figure 1~3
Figure 4~6
Figure 7~8
AI summary
Various embodiments relate to determining a parallel computation scheme for a neural network. A device may receive a computation graph and transform the computation graph into a dataflow graph comprising recursive subgraphs. Each recursive subgraph may comprise a tuple of another recursive subgraph and an operator node, or an empty graph. The device may determine a number of partitioning recursions based on a number of parallel computing devices. For each partitioning recursion, the device may determine costs corresponding to operator nodes, determine a processing order of the recursive subgraphs, and process the recursive subgraphs. To process a recursive subgraph, the device may select a partitioning axis for tensor(s) associated with an operator node of the recursive subgraph. The device may output a partitioning scheme comprising partitioning axes for each tensor associated with the operator nodes. Devices, methods, and computer programs are disclosed.