Distributed Neural Network Training for Accelerator Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The limited memory capacity of operation devices poses a challenge in training neural network models, particularly deep models, due to the large quantity of model parameters, leading to high storage pressure and difficulty in executing the training process.
Innovation Solution
Implementing a neural network model training method where different parts of the model parameters are stored across multiple accelerators, allowing each accelerator to obtain required parameters from others, reducing the memory needed and enabling efficient data parallel training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the complete model parameter is stored in a single device, then the neural network model training process can be executed, but the storage pressure on the device becomes excessively high
Solution Approach 1:
The complete model parameter is divided into multiple parts and distributed across multiple accelerators. Each accelerator stores only a portion of the model parameters (e.g., parameters for specific layers), rather than storing the entire model. This segmentation reduces the storage burden on each individual device while enabling the training process to proceed through coordinated computation across the distributed system.
2Quantity of substance
If model parameters are distributed across multiple accelerators, then storage pressure is reduced, but communication overhead and congestion may increase
Solution Approach 1:
The system performs preliminary actions by pre-loading and caching frequently accessed model parameters into local memory of each accelerator before they are needed for computation. This advance preparation reduces the need for repeated communication during the training process, as accelerators can retrieve parameters from their local memory rather than requesting them from remote storage during critical computation phases.
Solution Approach 2:
A parameter server or centralized storage system acts as an intermediary between the distributed accelerators. This mediator manages the distribution and retrieval of model parameters, optimizing communication patterns by batching requests, caching frequently accessed parameters, and coordinating data flow between accelerators to minimize communication congestion and latency.
3Quantity of substance
If each accelerator stores only local parameters, then memory capacity is sufficient, but obtaining complete model parameters requires additional operations
Solution Approach 1:
The system merges the computational capabilities of multiple accelerators with a centralized parameter management system. Each accelerator maintains its local parameters and performs computations, while the parameter server consolidates and manages the complete model parameters. This merging allows accelerators to efficiently combine their local computations with remotely stored parameters through standardized communication protocols, reducing the overall system complexity despite the distributed architecture.
Data Source
AI summary
The technology of this application relates to a neural network model training method, an image processing method, and an apparatus in the artificial intelligence. The training method includes each of at least one first accelerator training a neural network model based on at least one training sample. Before forward computation at an ith layer is performed, different parameters of the ith layer are obtained locally and from another accelerator, to obtain a complete model parameter of the ith layer. According to the method in this application, storage pressure of the first accelerator can be reduced.