Distributed Neural Network Training with Per-Model Dropout Noise
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional model parallelized neural networks waste calculation resources and are inefficient due to duplicate calculations in fully connected layers and excessive data communication, leading to reduced learning accuracy and increased time.
Innovation Solution
Implement a system where a neural network model is divided into distributed models, with only the convolution layer parallelized, and utilize a dropout layer with different noise settings for each model to optimize resource use and improve learning accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all layers including convolution layer and fully connected layer are divided and parallelized, then learning accuracy is maintained, but communication load increases and processing speed decreases due to frequent synchronization
Solution Approach 1:
The patent segments the neural network into two distinct parts: convolution layers that are parallelized across multiple processes, and fully connected layers that are not parallelized. This selective segmentation allows the system to benefit from parallelization in computationally intensive convolution operations while avoiding the communication overhead and accuracy degradation associated with parallelizing fully connected layers.
2Productivity
If only convolution layer is parallelized, then communication load is reduced, but calculation resources are wasted and processing becomes inefficient due to duplicate calculations in fully connected layers
Solution Approach 1:
The patent merges the execution of fully connected layers across all parallel processes, so that instead of each process independently executing the same fully connected layer calculations, the results are shared and utilized by all processes. This eliminates duplicate calculations in the fully connected layers while maintaining the parallelization benefits in the convolution layers.
3Productivity
If batch size is increased to achieve faster parallel processing, then processing speed increases, but learning accuracy is adversely affected
Solution Approach 1:
By segmenting the network into parallelizable convolution layers and non-parallelizable fully connected layers, the system can achieve speedup without increasing batch size. The convolution layers execute in parallel across multiple processes, providing computational speedup while maintaining the original batch size and thus preserving learning accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A non-transitory computer-readable recording medium storing a machine learning program of controlling machine learning of distributed neural network models generated by dividing a neural network, the machine learning program including instructions for causing a processor to execute processing including: adding, for each of the distributed neural network models, an individual noise for that distributed neural network model to a non-parallel processing block in that distributed neural network model such that the individual noise for that distributed neural network model is different from the individual noise for other distributed neural network models from among the distributed neural network models; and assigning, to a plurality of processes, the distributed neural network models added with the individual noise to cause each of the plurality of processes to perform the machine learning on an assigned neural network model from among the distributed neural network models added with the individual noise.