DNN Training Parallelism for Memory and Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training Deep Neural Networks (DNNs) face memory limitations due to the large memory requirements of weights and data, which restrict the size of DNNs that can be efficiently trained and lead to inefficient communication patterns and matrix multiplications, especially when using accelerators like GPUs.
Innovation Solution
A method that combines data and model parallelism by dividing neurons of a Convolutional Neural Network between multiple nodes, allocating mini-batches, and splitting them into sections for efficient matrix multiplication and communication, reducing the need for synchronization and optimizing GPU resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data distribution approach is used to train DNNs on multiple accelerators, then training can proceed with available memory, but the size of DNNs that can be efficiently trained is limited and batch sizes are restricted
Solution Approach 1:
The patent segments the DNN model into multiple parts distributed across different accelerators. Each accelerator stores and processes a specific portion of the model parameters, enabling training of larger models that exceed the memory capacity of individual accelerators. This segmentation allows the system to scale to larger model sizes while maintaining efficient training throughput.
Solution Approach 2:
The patent transitions from data parallelism to model parallelism, changing the dimension of distribution from data samples to model parameters. This dimensional shift enables training of larger models by distributing model weights across multiple accelerators rather than replicating entire models on each accelerator, thus overcoming the memory bottleneck while preserving training efficiency.
2Adaptability or versatility
If model distribution is applied across accelerators, then larger DNNs can be trained, but communication overhead increases and synchronization becomes more complex
Solution Approach 1:
The patent implements overlapping communication and computation by pre-fetching model parameters during idle periods. While one accelerator is computing gradients, other accelerators are simultaneously communicating updated parameters. This continuous useful action reduces idle waiting time and minimizes the impact of communication overhead on overall training performance.
Solution Approach 2:
The patent introduces a parameter server or coordination mechanism that acts as an intermediary to manage parameter distribution and synchronization across accelerators. This intermediary simplifies the communication protocol by centralizing coordination, reducing the complexity of direct peer-to-peer synchronization between multiple accelerators.
3Quantity of substance
If accelerator memory is limited, then hardware costs are reduced, but the memory requirements for training large DNNs cannot be met
Solution Approach 1:
The patent combines the memory resources of multiple accelerators by distributing model parameters across them. Each accelerator uses a portion of its memory to store specific model parameters, and the collective memory capacity of all accelerators enables training of large DNNs that would exceed the memory capacity of any single accelerator. This merging approach maintains cost-effectiveness while enabling large model training.
Data Source
AI summary
A computer-implemented method in a computing network of a number of processing nodes 1 to X, in the computing network neurons of a Convolutional Neural Network (CNN) are divided between the number of nodes. The method including allocating a mini-batch of input data from among mini-batches of input data to a node of the nodes; splitting the mini-batch into a number of mini-batch sections X corresponding and equal to the number of nodes; at the node retaining a mini-batch section which has a same number as the node and sending other mini-batch sections of the split mini-batch sections to corresponding other nodes according to a number of the split mini-batch sections; collating at the node the split mini-batch sections at the node into a single matrix and multiplying the collated matrix by the neurons to provide output data sections having one section of output data per each mini-batch. At the node sending the output data sections corresponding to the other corresponding nodes to the corresponding nodes and combining the output data sections in the node so that the node has output data for entire of the split mini-batch sections.