Neural Network Graph Pipelining for Parallel Batch Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph execution methods for neural network model computation suffer from low resource utilization and limited speedup and throughput due to synchronization, especially in large model computations, leading to inefficient execution of complex workloads.
Innovation Solution
A pipelining and parallelizing method is employed where training data is divided into batches, processed sequentially through forward and backward computation graphs, with each batch being executed on multiple executives using idle memory blocks to maintain pipeline balance and ensure synchronized parameter updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronization methods are used for graph execution, then model convergence is ensured, but resource utilization rate decreases and throughput is limited
Solution Approach 1:
The computation graph is segmented into multiple subgraphs that can be executed in parallel. Each subgraph processes a portion of the training data independently, allowing concurrent execution across multiple device processes while maintaining model convergence through coordinated parameter updates at fixed cycles.
Solution Approach 2:
Parameter updates are performed at fixed cycles before all mini-batches are fully processed. This preliminary action ensures model convergence is maintained while allowing parallel processing of multiple mini-batches simultaneously, improving resource utilization without sacrificing reliability.
2Reliability
If synchronization methods are used for graph execution, then parameter updates are coordinated, but speedup ratio is limited
Solution Approach 1:
The computation graph is divided into multiple subgraphs that can be executed in parallel. Each subgraph handles specific operations independently, enabling concurrent processing while maintaining parameter update coordination through fixed-cycle synchronization points.
Solution Approach 2:
Multiple mini-batches are processed continuously in parallel across different device processes without waiting for synchronization at each step. The useful action continues uninterrupted through parallel subgraph execution, with parameter updates coordinated only at fixed cycles, thereby increasing speedup ratio while maintaining coordination reliability.
3Productivity
If parallel processing of multiple mini-batches is increased, then throughput improves, but model convergence may be affected
Solution Approach 1:
Parameter updates are performed at fixed cycles before all parallel mini-batch processing is complete. This preliminary action ensures model convergence is maintained while allowing high throughput through continuous parallel processing of multiple mini-batches across different device processes.
Solution Approach 2:
Parameter updates occur periodically at fixed cycles rather than continuously. This periodic action allows multiple mini-batches to be processed in parallel simultaneously, improving throughput while maintaining model convergence through regular synchronization intervals that prevent excessive parallel processing effects.
4Productivity
If all device processes are kept busy without pipeline pause, then resource utilization improves, but pipeline balance becomes difficult to maintain
Solution Approach 1:
The computation graph is segmented into multiple subgraphs distributed across device processes. Each subgraph is designed to have comparable execution time, maintaining pipeline balance while keeping all device processes busy. The segmentation allows parallel execution without significant idle time, improving resource utilization.
Solution Approach 2:
The pipeline dynamically adjusts the distribution of mini-batches to different subgraphs based on execution progress. This dynamic balancing ensures all device processes remain busy while maintaining equilibrium in the pipeline, preventing bottlenecks and idle time through adaptive load distribution.
Data Source
AI summary
The present disclosure provides a pipelining and parallelizing graph execution method for neural network model computation and apparatus, and provides a pipelining and parallelizing graph execution method for neural network model computation and apparatus in a deep learning training system. The method includes the graph execution flow in a neural network model computation process and a process of cooperative work of all functional modules. The pipelining and parallelizing graph execution method for neural network model computation includes creating a graph executive on a native machine according to a physical computation graph compiled and generated by a deep learning framework.